[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:19 dist-test-slave-8w8g krb5kdc[6037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:19 dist-test-slave-8w8g krb5kdc[6037](info): set up 2 sockets
Sep 28 10:50:19 dist-test-slave-8w8g krb5kdc[6037](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 10:50:20.956650 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:21 dist-test-slave-8w8g krb5kdc[6037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:21.173475 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41293
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:21.520961 6050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:21.521324 6050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:21.521456 6050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:21.529068 6050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:21.529253 6050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:21.529359 6050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:21.529464 6050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:21.529539 6050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:21.541077 6050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:21.542531 6050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:21.543960 6050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:21.559252 6055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:21.559286 6058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:21.559314 6056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:21.560075 6050 server_base.cc:1034] running on GCE node
I20260928 10:50:21.561115 6050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:21.562937 6050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:21.564220 6050 hybrid_clock.cc:648] HybridClock initialized: now 1790592621564157 us; error 53 us; skew 500 ppm
I20260928 10:50:21.564669 6050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:21 dist-test-slave-8w8g krb5kdc[6037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592621, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:21.568293 6050 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:21.570720 6050 webserver.cc:460] Webserver started at http://127.5.227.190:33477/ using document root <none> and password file <none>
I20260928 10:50:21.572002 6050 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:21.572211 6050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:21.572651 6050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:21.577875 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "91cc49a4e489428f99ae46e0d076fb3d"
format_stamp: "Formatted at 2026-09-28 10:50:21 on dist-test-slave-8w8g"
server_key: "af4d1238706f92fb4ca0f22755eba2ed"
server_key_iv: "a98909a296a7225804413b86d99b387e"
server_key_version: "encryptionkey@0"
I20260928 10:50:21.579156 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "91cc49a4e489428f99ae46e0d076fb3d"
format_stamp: "Formatted at 2026-09-28 10:50:21 on dist-test-slave-8w8g"
server_key: "af4d1238706f92fb4ca0f22755eba2ed"
server_key_iv: "a98909a296a7225804413b86d99b387e"
server_key_version: "encryptionkey@0"
I20260928 10:50:21.585947 6050 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260928 10:50:21.590420 6065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:21.592374 6050 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:50:21.592623 6050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "91cc49a4e489428f99ae46e0d076fb3d"
format_stamp: "Formatted at 2026-09-28 10:50:21 on dist-test-slave-8w8g"
server_key: "af4d1238706f92fb4ca0f22755eba2ed"
server_key_iv: "a98909a296a7225804413b86d99b387e"
server_key_version: "encryptionkey@0"
I20260928 10:50:21.592953 6050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:21.632965 6050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:21.637815 6050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:21.638129 6050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:21.659132 6050 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:41293
I20260928 10:50:21.659164 6116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:41293 every 8 connection(s)
I20260928 10:50:21.660704 6050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:21.664803 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6050
I20260928 10:50:21.665184 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:21.665488 6117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:21.667898 6030 external_mini_cluster.cc:1428] Setting key 856738125a45b8d1668ad80d7fc188c7
I20260928 10:50:21.680250 6117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d: Bootstrap starting.
I20260928 10:50:21.684370 6117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:21.686133 6117 log.cc:824] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:21.690191 6117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d: No bootstrap required, opened a new log
I20260928 10:50:21.700224 6117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91cc49a4e489428f99ae46e0d076fb3d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41293 } }
I20260928 10:50:21.700757 6117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:21.701124 6117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91cc49a4e489428f99ae46e0d076fb3d, State: Initialized, Role: FOLLOWER
I20260928 10:50:21.702021 6117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91cc49a4e489428f99ae46e0d076fb3d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41293 } }
I20260928 10:50:21.702338 6117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:21.702507 6117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:21.702930 6117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 0 FOLLOWER]: Advancing to term 1
Sep 28 10:50:21 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:21.706462 6117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91cc49a4e489428f99ae46e0d076fb3d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41293 } }
I20260928 10:50:21.707223 6117 leader_election.cc:302] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91cc49a4e489428f99ae46e0d076fb3d; no voters:
I20260928 10:50:21.708145 6117 leader_election.cc:288] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:21.708319 6122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:21.711143 6117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:21.714512 6122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [term 1 LEADER]: Becoming Leader. State: Replica: 91cc49a4e489428f99ae46e0d076fb3d, State: Running, Role: LEADER
I20260928 10:50:21.716027 6122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91cc49a4e489428f99ae46e0d076fb3d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41293 } }
I20260928 10:50:21.719540 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:21.673203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.5.227.190:41293)
0928 10:50:21.674198 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:21.674218 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:21.675713 (+ 1495us) server_negotiation.cc:407] Connection header received
0928 10:50:21.686370 (+ 10657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:21.686407 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:21.687450 (+ 1043us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:21.688087 (+ 637us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:21.692014 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:21.693945 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:21.696826 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:21.698064 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:21.698384 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:21.706015 (+ 7631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:21.706027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:21.706045 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:21.706097 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:21.711216 (+ 5119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:21.712982 (+ 1766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:21.712999 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:21.713014 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:21.713372 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:21.714120 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:21.714130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:21.714136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:21.715059 (+ 923us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:21.715389 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:21.717045 (+ 1656us) server_negotiation.cc:299] Negotiation successful
0928 10:50:21.717442 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":437,"threads_started":1}
I20260928 10:50:21.720459 6123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91cc49a4e489428f99ae46e0d076fb3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91cc49a4e489428f99ae46e0d076fb3d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41293 } } }
I20260928 10:50:21.720376 6124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91cc49a4e489428f99ae46e0d076fb3d. Latest consensus state: current_term: 1 leader_uuid: "91cc49a4e489428f99ae46e0d076fb3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91cc49a4e489428f99ae46e0d076fb3d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41293 } } }
I20260928 10:50:21.721262 6123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:21.721268 6124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:21.722422 6133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:21.730067 6133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:21.741639 6133 catalog_manager.cc:1347] Generated new cluster ID: d002bd1b54664e6495c6bc077fcd4449
I20260928 10:50:21.741910 6133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:21.788898 6133 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:21.790972 6133 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:21.802621 6133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91cc49a4e489428f99ae46e0d076fb3d: Generated new TSK 0
I20260928 10:50:21.803869 6133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:50:21.872238 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41293
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:22.248944 6143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:22.249239 6143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:22.249398 6143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:22.261623 6143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:22.261799 6143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:22.261977 6143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:50:22.272491 6143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:22.273895 6143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:22.275266 6143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:22.289556 6149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:22.289573 6151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:22.289624 6148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:22.290632 6143 server_base.cc:1034] running on GCE node
I20260928 10:50:22.291311 6143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:22.292483 6143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:22.293723 6143 hybrid_clock.cc:648] HybridClock initialized: now 1790592622293665 us; error 46 us; skew 500 ppm
I20260928 10:50:22.294094 6143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:22 dist-test-slave-8w8g krb5kdc[6037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592622, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:22.297634 6143 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:50:22.299597 6143 webserver.cc:460] Webserver started at http://127.5.227.129:37973/ using document root <none> and password file <none>
I20260928 10:50:22.300405 6143 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:22.300514 6143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:22.300801 6143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:22.303921 6143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "ae4a2508486e455ca66e7c1a3f2666e7"
format_stamp: "Formatted at 2026-09-28 10:50:22 on dist-test-slave-8w8g"
server_key: "98ffe019be21bbe1e0873a1aecc019d6"
server_key_iv: "f7825a3ffc0176171a3d0231740592e4"
server_key_version: "encryptionkey@0"
I20260928 10:50:22.304669 6143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "ae4a2508486e455ca66e7c1a3f2666e7"
format_stamp: "Formatted at 2026-09-28 10:50:22 on dist-test-slave-8w8g"
server_key: "98ffe019be21bbe1e0873a1aecc019d6"
server_key_iv: "f7825a3ffc0176171a3d0231740592e4"
server_key_version: "encryptionkey@0"
I20260928 10:50:22.310619 6143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:50:22.314543 6158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:22.316205 6143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:50:22.316398 6143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "ae4a2508486e455ca66e7c1a3f2666e7"
format_stamp: "Formatted at 2026-09-28 10:50:22 on dist-test-slave-8w8g"
server_key: "98ffe019be21bbe1e0873a1aecc019d6"
server_key_iv: "f7825a3ffc0176171a3d0231740592e4"
server_key_version: "encryptionkey@0"
I20260928 10:50:22.316569 6143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:22.343982 6143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:22.349102 6143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:22.349478 6143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:22.350889 6143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:22.353096 6143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:22.353206 6143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:22.353348 6143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:22.353412 6143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:22.396556 6143 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:39819
I20260928 10:50:22.396566 6270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:39819 every 8 connection(s)
I20260928 10:50:22.398207 6143 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:50:22.402724 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6143
I20260928 10:50:22.403025 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:50:22.403882 6030 external_mini_cluster.cc:1428] Setting key b2d5ca33940b91cbcaad1030c6ea33fc
Sep 28 10:50:22 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592622, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:50:22.422744 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:22.401332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:57529 (local address 127.5.227.190:41293)
0928 10:50:22.401913 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:22.401918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:22.403085 (+ 1167us) server_negotiation.cc:407] Connection header received
0928 10:50:22.405329 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:22.405335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:22.405445 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:22.405567 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:22.407382 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.408864 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.410735 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.411143 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.411302 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:22.416051 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:22.416058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:22.416061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:22.416101 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:22.419314 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:22.420228 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:22.420235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:22.420238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:22.420372 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:22.420820 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:22.420833 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:22.420835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:22.421162 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:22.421327 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:22.422122 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 10:50:22.422361 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":284,"threads_started":1}
I20260928 10:50:22.423935 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:22.401968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41293 (local address 127.5.227.129:57529)
0928 10:50:22.402875 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:22.402932 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:22.404984 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:22.405835 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:22.405844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:22.406532 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:22.407155 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:22.407175 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.409098 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.409104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:22.410528 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:22.410548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.411446 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.411454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:22.411713 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:22.412969 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:22.412995 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:22.415707 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:22.419525 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:22.419534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:22.419550 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:22.420078 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:22.420494 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:22.420499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:22.420505 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:22.420693 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:22.421298 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:22.421307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:22.421692 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 10:50:22.421974 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 10:50:22.422546 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":404,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:50:22.426146 6271 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41293
I20260928 10:50:22.426529 6271 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:22.427335 6271 heartbeater.cc:511] Master 127.5.227.190:41293 requested a full tablet report, sending...
I20260928 10:50:22.430020 6082 ts_manager.cc:194] Registered new tserver with Master: ae4a2508486e455ca66e7c1a3f2666e7 (127.5.227.129:39819)
I20260928 10:50:22.432454 6082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:57529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:50:22.452556 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41293
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:22.810664 6277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:22.810968 6277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:22.811067 6277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:22.819429 6277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:22.819605 6277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:22.819832 6277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:50:22.830426 6277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:22.831812 6277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:22.833204 6277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:22.848001 6282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:22.847960 6283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:22.847923 6285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:22.849025 6277 server_base.cc:1034] running on GCE node
I20260928 10:50:22.850014 6277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:22.851148 6277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:22.852385 6277 hybrid_clock.cc:648] HybridClock initialized: now 1790592622852299 us; error 66 us; skew 500 ppm
I20260928 10:50:22.852779 6277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:22 dist-test-slave-8w8g krb5kdc[6037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592622, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:22.857148 6277 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:50:22.859278 6277 webserver.cc:460] Webserver started at http://127.5.227.130:35339/ using document root <none> and password file <none>
I20260928 10:50:22.860189 6277 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:22.860359 6277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:22.860705 6277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:22.864004 6277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "b6ae513cc0584046a7e2cd8ffe3f757a"
format_stamp: "Formatted at 2026-09-28 10:50:22 on dist-test-slave-8w8g"
server_key: "5a04d65f4300078707f79868745b94b0"
server_key_iv: "6a80ea057c855485b0a11f48bcce518b"
server_key_version: "encryptionkey@0"
I20260928 10:50:22.864913 6277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "b6ae513cc0584046a7e2cd8ffe3f757a"
format_stamp: "Formatted at 2026-09-28 10:50:22 on dist-test-slave-8w8g"
server_key: "5a04d65f4300078707f79868745b94b0"
server_key_iv: "6a80ea057c855485b0a11f48bcce518b"
server_key_version: "encryptionkey@0"
I20260928 10:50:22.870976 6277 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 10:50:22.875422 6292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:22.877271 6277 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:50:22.877529 6277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "b6ae513cc0584046a7e2cd8ffe3f757a"
format_stamp: "Formatted at 2026-09-28 10:50:22 on dist-test-slave-8w8g"
server_key: "5a04d65f4300078707f79868745b94b0"
server_key_iv: "6a80ea057c855485b0a11f48bcce518b"
server_key_version: "encryptionkey@0"
I20260928 10:50:22.877792 6277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:22.894658 6277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:22.901592 6277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:22.902184 6277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:22.903496 6277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:22.905519 6277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:22.905663 6277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:22.906044 6277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:22.906154 6277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:22.947471 6277 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:38547
I20260928 10:50:22.947516 6404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:38547 every 8 connection(s)
I20260928 10:50:22.948894 6277 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:50:22.952852 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6277
I20260928 10:50:22.953109 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:50:22.953683 6030 external_mini_cluster.cc:1428] Setting key 702efc75692a2dad2dddb2425e71be9a
Sep 28 10:50:22 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592622, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:50:22.973495 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:22.951640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:55551 (local address 127.5.227.190:41293)
0928 10:50:22.952116 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:22.952120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:22.953938 (+ 1818us) server_negotiation.cc:407] Connection header received
0928 10:50:22.956304 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:22.956309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:22.956400 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:22.956535 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:22.958231 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.958999 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.960784 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.961235 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.961385 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:22.965681 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:22.965687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:22.965691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:22.965731 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:22.969269 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:22.970538 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:22.970543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:22.970546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:22.970709 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:22.971261 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:22.971265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:22.971267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:22.971571 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:22.971785 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:22.972824 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0928 10:50:22.973095 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":273,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:50:22.974727 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:22.952224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41293 (local address 127.5.227.130:55551)
0928 10:50:22.953689 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:22.953786 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:22.955989 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:22.956830 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:22.956841 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:22.957402 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:22.958029 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:22.958048 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.959194 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.959199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:22.960611 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:22.960626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:22.961486 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:22.961493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:22.961757 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:22.962552 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:22.962587 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:22.965348 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:22.969599 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:22.969613 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:22.969635 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:22.970338 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:22.970879 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:22.970884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:22.970886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:22.971095 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:22.971759 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:22.971769 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:22.972205 (+ 436us) client_negotiation.cc:769] Sending connection context
0928 10:50:22.972706 (+ 501us) client_negotiation.cc:240] Negotiation successful
0928 10:50:22.973326 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":974,"threads_started":1}
I20260928 10:50:22.976495 6405 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41293
I20260928 10:50:22.976902 6405 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:22.977885 6405 heartbeater.cc:511] Master 127.5.227.190:41293 requested a full tablet report, sending...
I20260928 10:50:22.980418 6082 ts_manager.cc:194] Registered new tserver with Master: b6ae513cc0584046a7e2cd8ffe3f757a (127.5.227.130:38547)
I20260928 10:50:22.981364 6082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:55551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:50:23.009743 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41293
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:23.380339 6411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:23.380681 6411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:23.380781 6411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:23.388532 6411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:23.388657 6411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:23.388783 6411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:50:23.401823 6411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:23.403338 6411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:23.404755 6411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:23.419615 6416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:23.420536 6419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:23.420603 6417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:23.420964 6411 server_base.cc:1034] running on GCE node
I20260928 10:50:23.421689 6411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:23.422709 6411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:23.424013 6411 hybrid_clock.cc:648] HybridClock initialized: now 1790592623423932 us; error 70 us; skew 500 ppm
I20260928 10:50:23.424412 6411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:23.428284 6411 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:50:23.430219 6411 webserver.cc:460] Webserver started at http://127.5.227.131:41331/ using document root <none> and password file <none>
I20260928 10:50:23.431041 6411 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:23.431231 6411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:23.431607 6411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:23.435084 6411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "3654b838d8964eccb7fdf2d44696f2ca"
format_stamp: "Formatted at 2026-09-28 10:50:23 on dist-test-slave-8w8g"
server_key: "2ca1a9d1444b181a1d12fe53cd29bd41"
server_key_iv: "45dfb19ffbdd3453e9c0af6c61926aa7"
server_key_version: "encryptionkey@0"
I20260928 10:50:23.436136 6411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "3654b838d8964eccb7fdf2d44696f2ca"
format_stamp: "Formatted at 2026-09-28 10:50:23 on dist-test-slave-8w8g"
server_key: "2ca1a9d1444b181a1d12fe53cd29bd41"
server_key_iv: "45dfb19ffbdd3453e9c0af6c61926aa7"
server_key_version: "encryptionkey@0"
I20260928 10:50:23.436424 6271 heartbeater.cc:503] Master 127.5.227.190:41293 was elected leader, sending a full tablet report...
I20260928 10:50:23.443310 6411 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:50:23.447443 6426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:23.449326 6411 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 10:50:23.449541 6411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "3654b838d8964eccb7fdf2d44696f2ca"
format_stamp: "Formatted at 2026-09-28 10:50:23 on dist-test-slave-8w8g"
server_key: "2ca1a9d1444b181a1d12fe53cd29bd41"
server_key_iv: "45dfb19ffbdd3453e9c0af6c61926aa7"
server_key_version: "encryptionkey@0"
I20260928 10:50:23.449766 6411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:23.484563 6411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:23.490186 6411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:23.490468 6411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:23.491531 6411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:23.493238 6411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:23.493319 6411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:23.493407 6411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:23.493459 6411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:23.535255 6411 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:36369
I20260928 10:50:23.535431 6538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:36369 every 8 connection(s)
I20260928 10:50:23.536933 6411 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:50:23.541563 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6411
I20260928 10:50:23.541771 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:50:23.542340 6030 external_mini_cluster.cc:1428] Setting key 068b83fb6e6132303738d479e703976b
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:23.562477 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:23.540028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:58367 (local address 127.5.227.190:41293)
0928 10:50:23.540556 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:23.540561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:23.541709 (+ 1148us) server_negotiation.cc:407] Connection header received
0928 10:50:23.544301 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:23.544306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:23.544420 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:23.544569 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:23.547094 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.548007 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.549862 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.550344 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.550457 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:23.554385 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:23.554390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:23.554393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:23.554431 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:23.558050 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.559520 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.559526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.559529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.559724 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.560417 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.560424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.560428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.560955 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:23.561091 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:23.562003 (+ 912us) server_negotiation.cc:299] Negotiation successful
0928 10:50:23.562198 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":272,"threads_started":1}
I20260928 10:50:23.563529 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:23.540453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41293 (local address 127.5.227.131:58367)
0928 10:50:23.541452 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:23.541510 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:23.543957 (+ 2447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:23.545091 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:23.545107 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:23.546034 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:23.546878 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:23.546903 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.548204 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.548209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:23.549620 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:23.549635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.550584 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.550589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:23.550884 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:23.551656 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:23.551687 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:23.554042 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:23.558291 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:23.558301 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:23.558323 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:23.559342 (+ 1019us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:23.560015 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:23.560025 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:23.560027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:23.560247 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:23.561133 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:23.561145 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:23.561557 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 10:50:23.561849 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 10:50:23.562288 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":447,"threads_started":1}
I20260928 10:50:23.565445 6539 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41293
I20260928 10:50:23.565912 6539 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:23.566797 6539 heartbeater.cc:511] Master 127.5.227.190:41293 requested a full tablet report, sending...
I20260928 10:50:23.568816 6082 ts_manager.cc:194] Registered new tserver with Master: 3654b838d8964eccb7fdf2d44696f2ca (127.5.227.131:36369)
I20260928 10:50:23.570302 6082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:58367
I20260928 10:50:23.574469 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:23.630729 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:23.609435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.5.227.129:39819)
0928 10:50:23.610219 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:23.610234 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:23.610262 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:50:23.610326 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:23.610333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:23.610581 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:23.610789 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:23.613706 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.615199 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.617828 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.618561 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.618875 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:23.624090 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:23.624106 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:23.624123 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:23.624181 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:23.627476 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.628411 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.628419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.628422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.628564 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.629014 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.629019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.629021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.629390 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:23.629709 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:23.630045 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 10:50:23.630359 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":384,"threads_started":1}
W20260928 10:50:23.632810 6186 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36682
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:23.684407 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:23.667365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.5.227.129:39819)
0928 10:50:23.667631 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:23.667637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:23.668047 (+ 410us) server_negotiation.cc:407] Connection header received
0928 10:50:23.668336 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:23.668340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:23.668443 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:23.668553 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:23.670130 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.670982 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.672373 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.672800 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.672906 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:23.677490 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:23.677495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:23.677499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:23.677540 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:23.680686 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.681644 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.681649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.681652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.681850 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.682467 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.682472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.682475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.682914 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:23.683136 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:23.683716 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 10:50:23.684009 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:50:23.686237 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6143
I20260928 10:50:23.701840 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:23.687464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.5.227.129:39819)
0928 10:50:23.687642 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:23.687646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:23.688248 (+ 602us) server_negotiation.cc:407] Connection header received
0928 10:50:23.688397 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:23.688401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:23.688467 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:23.688566 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:23.690042 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.690775 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.692437 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.692918 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.693025 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:23.696071 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:23.696076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:23.696079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:23.696115 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:23.699165 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.699912 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.699917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.699919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.700058 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.700481 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.700485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.700487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.700787 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:23.701016 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:23.701375 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 10:50:23.701547 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:50:23.702765 6266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:23.789079 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6143
I20260928 10:50:23.808346 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6277
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:50:23.829413 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:23.809543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.5.227.130:38547)
0928 10:50:23.810240 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:23.810250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:23.810281 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:50:23.810350 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:23.810357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:23.810593 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:23.810864 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:23.812918 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.813957 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.815285 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.815966 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.816117 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:23.821878 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:23.821895 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:23.821912 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:23.821958 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:23.825871 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.826809 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.826819 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.826822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.826968 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.827471 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.827477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.827482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.827961 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:23.828324 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:23.828685 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 10:50:23.828989 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":377,"threads_started":1}
I20260928 10:50:23.831151 6400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:23.926268 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6277
I20260928 10:50:23.953202 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6411
Sep 28 10:50:23 dist-test-slave-8w8g krb5kdc[6037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:50:23.977586 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:23.958825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.5.227.131:36369)
0928 10:50:23.959416 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:23.959424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:23.959457 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:50:23.959522 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:23.959529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:23.959974 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:23.960198 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:23.961770 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.963408 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.965023 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:23.965622 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:23.965752 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:23.970335 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:23.970352 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:23.970368 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:23.970411 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:23.974190 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.975008 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.975023 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.975026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.975175 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:23.975579 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:23.975582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:23.975584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:23.976048 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:23.976297 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:23.976705 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 10:50:23.977093 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":277,"threads_started":1}
I20260928 10:50:23.979341 6534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:24.064572 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6411
I20260928 10:50:24.084344 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6050
I20260928 10:50:24.086117 6112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:24.192955 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6050
2026-09-28T10:50:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4426 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:24 dist-test-slave-8w8g krb5kdc[6561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:24 dist-test-slave-8w8g krb5kdc[6561](info): set up 2 sockets
Sep 28 10:50:24 dist-test-slave-8w8g krb5kdc[6561](info): commencing operation
krb5kdc: starting...
W20260928 10:50:25.336308 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:25 dist-test-slave-8w8g krb5kdc[6561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:25.537927 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:46581
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:46581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:25.925928 6574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:25.927018 6574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:25.927273 6574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:25.935956 6574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:25.936110 6574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:25.936161 6574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:25.936216 6574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:25.936267 6574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:25.948794 6574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:46581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:25.950217 6574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:25.951511 6574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:25.965512 6582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:25.965534 6579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:25.965514 6580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:25.966995 6574 server_base.cc:1034] running on GCE node
I20260928 10:50:25.968020 6574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:25.969936 6574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:25.971203 6574 hybrid_clock.cc:648] HybridClock initialized: now 1790592625971147 us; error 49 us; skew 500 ppm
I20260928 10:50:25.971662 6574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:25 dist-test-slave-8w8g krb5kdc[6561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592625, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:25.983879 6574 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:25.985785 6574 webserver.cc:460] Webserver started at http://127.5.227.190:35727/ using document root <none> and password file <none>
I20260928 10:50:25.986598 6574 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:25.986775 6574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:25.987123 6574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:25.990669 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "e66c07a3b36e4701a5ef656dc083473e"
format_stamp: "Formatted at 2026-09-28 10:50:25 on dist-test-slave-8w8g"
server_key: "7267d63ebfd5de6e0bc9b6d8f8a396ca"
server_key_iv: "1c68832e42c5c264a9adb3635ade7f80"
server_key_version: "encryptionkey@0"
I20260928 10:50:25.991782 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "e66c07a3b36e4701a5ef656dc083473e"
format_stamp: "Formatted at 2026-09-28 10:50:25 on dist-test-slave-8w8g"
server_key: "7267d63ebfd5de6e0bc9b6d8f8a396ca"
server_key_iv: "1c68832e42c5c264a9adb3635ade7f80"
server_key_version: "encryptionkey@0"
I20260928 10:50:25.999116 6574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260928 10:50:26.004542 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:26.006717 6574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:50:26.007025 6574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "e66c07a3b36e4701a5ef656dc083473e"
format_stamp: "Formatted at 2026-09-28 10:50:25 on dist-test-slave-8w8g"
server_key: "7267d63ebfd5de6e0bc9b6d8f8a396ca"
server_key_iv: "1c68832e42c5c264a9adb3635ade7f80"
server_key_version: "encryptionkey@0"
I20260928 10:50:26.007383 6574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:26.022979 6574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:26.029448 6574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:26.029817 6574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:26.053362 6574 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:46581
I20260928 10:50:26.053381 6640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:46581 every 8 connection(s)
I20260928 10:50:26.055050 6574 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:26.060204 6641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:26.061051 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6574
I20260928 10:50:26.061349 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:26.062430 6030 external_mini_cluster.cc:1428] Setting key 584dfc1495fff44421e39cf2d289bce0
I20260928 10:50:26.078150 6641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e: Bootstrap starting.
Sep 28 10:50:26 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:26.082901 6641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:26.084444 6641 log.cc:824] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:26.088351 6641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e: No bootstrap required, opened a new log
I20260928 10:50:26.092980 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:26.065341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.5.227.190:46581)
0928 10:50:26.066302 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:26.066320 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:26.066376 (+ 56us) server_negotiation.cc:407] Connection header received
0928 10:50:26.068372 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:26.068405 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:26.068933 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:26.069359 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:26.071338 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:26.073412 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:26.075821 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:26.076703 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:26.076895 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:26.083013 (+ 6118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:26.083024 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:26.083042 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:26.083121 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:26.087006 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:26.088257 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:26.088272 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:26.088282 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:26.088586 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:26.089196 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:26.089202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:26.089206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:26.089971 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:26.090506 (+ 535us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:26.091112 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 10:50:26.091617 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":439,"threads_started":1}
I20260928 10:50:26.098402 6641 raft_consensus.cc:348] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66c07a3b36e4701a5ef656dc083473e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46581 } }
I20260928 10:50:26.098910 6641 raft_consensus.cc:374] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:26.099045 6641 raft_consensus.cc:729] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66c07a3b36e4701a5ef656dc083473e, State: Initialized, Role: FOLLOWER
I20260928 10:50:26.099869 6641 consensus_queue.cc:260] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66c07a3b36e4701a5ef656dc083473e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46581 } }
I20260928 10:50:26.100172 6641 raft_consensus.cc:388] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:26.100350 6641 raft_consensus.cc:482] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:26.100596 6641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:26.103152 6641 raft_consensus.cc:504] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66c07a3b36e4701a5ef656dc083473e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46581 } }
I20260928 10:50:26.103859 6641 leader_election.cc:302] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e66c07a3b36e4701a5ef656dc083473e; no voters:
I20260928 10:50:26.104701 6641 leader_election.cc:288] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:26.105109 6646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:26.106101 6646 raft_consensus.cc:686] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [term 1 LEADER]: Becoming Leader. State: Replica: e66c07a3b36e4701a5ef656dc083473e, State: Running, Role: LEADER
I20260928 10:50:26.106884 6646 consensus_queue.cc:237] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66c07a3b36e4701a5ef656dc083473e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46581 } }
I20260928 10:50:26.107581 6641 sys_catalog.cc:564] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:26.111929 6648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e66c07a3b36e4701a5ef656dc083473e. Latest consensus state: current_term: 1 leader_uuid: "e66c07a3b36e4701a5ef656dc083473e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66c07a3b36e4701a5ef656dc083473e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46581 } } }
I20260928 10:50:26.112440 6648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:26.112244 6647 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e66c07a3b36e4701a5ef656dc083473e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66c07a3b36e4701a5ef656dc083473e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46581 } } }
I20260928 10:50:26.112892 6647 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:26.115509 6656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:26.121340 6656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:26.132813 6656 catalog_manager.cc:1347] Generated new cluster ID: 0548e506578d4e2aba35ac4f3dafb811
I20260928 10:50:26.132993 6656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:26.160142 6656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:26.162554 6656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:26.172883 6656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e66c07a3b36e4701a5ef656dc083473e: Generated new TSK 0
I20260928 10:50:26.174197 6656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:50:26.256646 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46581
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:26.688846 6667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:26.689195 6667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:26.689365 6667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:26.697409 6667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:26.697561 6667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:26.697769 6667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:50:26.709313 6667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:26.710836 6667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:26.712679 6667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:26.730041 6673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:26.730216 6675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:26.730374 6672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:26.732015 6667 server_base.cc:1034] running on GCE node
I20260928 10:50:26.733140 6667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:26.734500 6667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:26.735970 6667 hybrid_clock.cc:648] HybridClock initialized: now 1790592626735918 us; error 41 us; skew 500 ppm
I20260928 10:50:26.736413 6667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:26 dist-test-slave-8w8g krb5kdc[6561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592626, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:26.740525 6667 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:50:26.742508 6667 webserver.cc:460] Webserver started at http://127.5.227.129:33605/ using document root <none> and password file <none>
I20260928 10:50:26.743377 6667 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:26.743510 6667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:26.743990 6667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:26.748023 6667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "886e02cbd99e489d92d5725eab22cb66"
format_stamp: "Formatted at 2026-09-28 10:50:26 on dist-test-slave-8w8g"
server_key: "23d5074324a61dd89f825906a3b04c53"
server_key_iv: "af9594006bc3b4c0406ab2f64cc4f005"
server_key_version: "encryptionkey@0"
I20260928 10:50:26.749066 6667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "886e02cbd99e489d92d5725eab22cb66"
format_stamp: "Formatted at 2026-09-28 10:50:26 on dist-test-slave-8w8g"
server_key: "23d5074324a61dd89f825906a3b04c53"
server_key_iv: "af9594006bc3b4c0406ab2f64cc4f005"
server_key_version: "encryptionkey@0"
I20260928 10:50:26.756130 6667 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 10:50:26.760835 6682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:26.762851 6667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:50:26.763056 6667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "886e02cbd99e489d92d5725eab22cb66"
format_stamp: "Formatted at 2026-09-28 10:50:26 on dist-test-slave-8w8g"
server_key: "23d5074324a61dd89f825906a3b04c53"
server_key_iv: "af9594006bc3b4c0406ab2f64cc4f005"
server_key_version: "encryptionkey@0"
I20260928 10:50:26.763377 6667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:26.783989 6667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:26.791015 6667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:26.791577 6667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:26.793051 6667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:26.794595 6667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:26.794677 6667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:26.794813 6667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:26.794873 6667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:26.838423 6667 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:38415
I20260928 10:50:26.838760 6794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:38415 every 8 connection(s)
I20260928 10:50:26.840019 6667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:50:26.843411 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6667
I20260928 10:50:26.843837 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:50:26.845073 6030 external_mini_cluster.cc:1428] Setting key 09ff2d690e8c37f2b5a8732c899a6679
Sep 28 10:50:26 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592626, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:50:26.870788 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:26.843186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:49711 (local address 127.5.227.190:46581)
0928 10:50:26.844326 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:26.844334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:26.846223 (+ 1889us) server_negotiation.cc:407] Connection header received
0928 10:50:26.849956 (+ 3733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:26.849965 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:26.850073 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:26.850216 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:26.852410 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:26.853320 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:26.856698 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:26.857151 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:26.857283 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:26.862565 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:26.862576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:26.862584 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:26.862646 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:26.866156 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:26.867243 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:26.867249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:26.867253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:26.867499 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:26.868338 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:26.868347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:26.868350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:26.868743 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:26.868901 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:26.870191 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0928 10:50:26.870421 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":551,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:50:26.872272 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:26.844334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46581 (local address 127.5.227.129:49711)
0928 10:50:26.845670 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:26.845729 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:26.849526 (+ 3797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:26.850550 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:26.850561 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:26.851246 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:26.852171 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:26.852196 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:26.853552 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:26.853560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:26.856409 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:26.856437 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:26.857473 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:26.857485 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:26.858015 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:26.858946 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:26.858992 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:26.862190 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:26.866394 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:26.866408 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:26.866434 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:26.867020 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:26.867860 (+ 840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:26.867865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:26.867867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:26.868109 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:26.868930 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:26.868938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:26.869384 (+ 446us) client_negotiation.cc:769] Sending connection context
0928 10:50:26.869960 (+ 576us) client_negotiation.cc:240] Negotiation successful
0928 10:50:26.870532 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":595,"threads_started":1}
I20260928 10:50:26.874648 6795 heartbeater.cc:347] Connected to a master server at 127.5.227.190:46581
I20260928 10:50:26.875044 6795 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:26.876183 6795 heartbeater.cc:511] Master 127.5.227.190:46581 requested a full tablet report, sending...
I20260928 10:50:26.878942 6606 ts_manager.cc:194] Registered new tserver with Master: 886e02cbd99e489d92d5725eab22cb66 (127.5.227.129:38415)
I20260928 10:50:26.880620 6606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:49711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:50:26.910068 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46581
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:27.313869 6801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:27.314235 6801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:27.314410 6801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:27.322527 6801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:27.322713 6801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:27.322906 6801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:50:27.334940 6801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:27.336520 6801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:27.337960 6801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:27.352485 6806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:27.352658 6807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:27.353029 6809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:27.353863 6801 server_base.cc:1034] running on GCE node
I20260928 10:50:27.354660 6801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:27.355880 6801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:27.357209 6801 hybrid_clock.cc:648] HybridClock initialized: now 1790592627357115 us; error 45 us; skew 500 ppm
I20260928 10:50:27.357580 6801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:27 dist-test-slave-8w8g krb5kdc[6561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592627, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:27.360984 6801 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:50:27.362562 6801 webserver.cc:460] Webserver started at http://127.5.227.130:39435/ using document root <none> and password file <none>
I20260928 10:50:27.363380 6801 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:27.363520 6801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:27.363965 6801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:27.366986 6801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "58c9ef76b7584606a49000ec67c69a54"
format_stamp: "Formatted at 2026-09-28 10:50:27 on dist-test-slave-8w8g"
server_key: "59b3f3eecc60a2e7746510e656a9019c"
server_key_iv: "e1b6c70ab0c7164779063b33f68b12c1"
server_key_version: "encryptionkey@0"
I20260928 10:50:27.367856 6801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "58c9ef76b7584606a49000ec67c69a54"
format_stamp: "Formatted at 2026-09-28 10:50:27 on dist-test-slave-8w8g"
server_key: "59b3f3eecc60a2e7746510e656a9019c"
server_key_iv: "e1b6c70ab0c7164779063b33f68b12c1"
server_key_version: "encryptionkey@0"
I20260928 10:50:27.374477 6801 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:50:27.378356 6816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:27.380108 6801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 10:50:27.380321 6801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "58c9ef76b7584606a49000ec67c69a54"
format_stamp: "Formatted at 2026-09-28 10:50:27 on dist-test-slave-8w8g"
server_key: "59b3f3eecc60a2e7746510e656a9019c"
server_key_iv: "e1b6c70ab0c7164779063b33f68b12c1"
server_key_version: "encryptionkey@0"
I20260928 10:50:27.380548 6801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:27.402477 6801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:27.408929 6801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:27.409217 6801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:27.410264 6801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:27.412022 6801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:27.412101 6801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:27.412305 6801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:27.412371 6801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:27.449591 6801 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:43303
I20260928 10:50:27.449661 6928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:43303 every 8 connection(s)
I20260928 10:50:27.451150 6801 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:50:27.455482 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6801
I20260928 10:50:27.455767 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:50:27.456358 6030 external_mini_cluster.cc:1428] Setting key 7399d9c4e64a88cd5e4f3acc7c832bb6
Sep 28 10:50:27 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592627, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:50:27.473464 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:27.454225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:59949 (local address 127.5.227.190:46581)
0928 10:50:27.454814 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:27.454819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:27.456038 (+ 1219us) server_negotiation.cc:407] Connection header received
0928 10:50:27.458292 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:27.458298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:27.458382 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:27.458513 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:27.460455 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:27.461207 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:27.462847 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:27.463291 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:27.463434 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:27.467122 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:27.467127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:27.467130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:27.467165 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:27.470059 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:27.470907 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:27.470920 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:27.470924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:27.471053 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:27.471555 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:27.471566 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:27.471568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:27.471918 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:27.472080 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:27.472865 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 10:50:27.473096 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260928 10:50:27.474556 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:27.454908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46581 (local address 127.5.227.130:59949)
0928 10:50:27.455847 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:27.455897 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:27.457979 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:27.458837 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:27.458847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:27.459397 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:27.460249 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:27.460268 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:27.461388 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:27.461393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:27.462679 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:27.462691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:27.463552 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:27.463560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:27.463871 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:27.464624 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:27.464651 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:27.466854 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:27.470245 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:27.470256 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:27.470277 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:27.470759 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:27.471177 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:27.471183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:27.471186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:27.471411 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:27.472054 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:27.472064 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:27.472464 (+ 400us) client_negotiation.cc:769] Sending connection context
0928 10:50:27.472763 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 10:50:27.473366 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":387,"threads_started":1}
I20260928 10:50:27.476624 6929 heartbeater.cc:347] Connected to a master server at 127.5.227.190:46581
I20260928 10:50:27.477114 6929 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:27.477972 6929 heartbeater.cc:511] Master 127.5.227.190:46581 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:50:27.479705 6606 ts_manager.cc:194] Registered new tserver with Master: 58c9ef76b7584606a49000ec67c69a54 (127.5.227.130:43303)
I20260928 10:50:27.480636 6606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:59949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:50:27.512729 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46581
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:50:27.885839 6795 heartbeater.cc:503] Master 127.5.227.190:46581 was elected leader, sending a full tablet report...
W20260928 10:50:27.938023 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:27.938351 6935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:27.938452 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:27.948254 6935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:27.948412 6935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:27.948547 6935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:50:27.960937 6935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:27.962709 6935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:27.964509 6935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:27.982154 6941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:27.982334 6940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:27.982791 6943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:27.983124 6935 server_base.cc:1034] running on GCE node
I20260928 10:50:27.983996 6935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:27.985534 6935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:27.986977 6935 hybrid_clock.cc:648] HybridClock initialized: now 1790592627986876 us; error 71 us; skew 500 ppm
I20260928 10:50:27.987408 6935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:27 dist-test-slave-8w8g krb5kdc[6561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592627, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:27.991101 6935 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:50:27.992817 6935 webserver.cc:460] Webserver started at http://127.5.227.131:38421/ using document root <none> and password file <none>
I20260928 10:50:27.993578 6935 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:27.993695 6935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:27.993978 6935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:27.997251 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "d467b3da58f94938b38d0dd797999c98"
format_stamp: "Formatted at 2026-09-28 10:50:27 on dist-test-slave-8w8g"
server_key: "5c3a0131c469203ac7f87a84662a62df"
server_key_iv: "a11e6841d369597eb5a87e139f38eee9"
server_key_version: "encryptionkey@0"
I20260928 10:50:27.998088 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "d467b3da58f94938b38d0dd797999c98"
format_stamp: "Formatted at 2026-09-28 10:50:27 on dist-test-slave-8w8g"
server_key: "5c3a0131c469203ac7f87a84662a62df"
server_key_iv: "a11e6841d369597eb5a87e139f38eee9"
server_key_version: "encryptionkey@0"
I20260928 10:50:28.006435 6935 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260928 10:50:28.012451 6950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:28.014561 6935 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260928 10:50:28.014808 6935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "d467b3da58f94938b38d0dd797999c98"
format_stamp: "Formatted at 2026-09-28 10:50:27 on dist-test-slave-8w8g"
server_key: "5c3a0131c469203ac7f87a84662a62df"
server_key_iv: "a11e6841d369597eb5a87e139f38eee9"
server_key_version: "encryptionkey@0"
I20260928 10:50:28.015017 6935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:28.060123 6935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:28.065833 6935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:28.066188 6935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:28.067271 6935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:28.069088 6935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:28.069340 6935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:28.069525 6935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:28.069628 6935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:28.112021 6935 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:43681
I20260928 10:50:28.112048 7062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:43681 every 8 connection(s)
I20260928 10:50:28.113616 6935 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:50:28.118194 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 6935
I20260928 10:50:28.118426 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:50:28.118902 6030 external_mini_cluster.cc:1428] Setting key 76102b1bee430a10edd250ae4c0048f5
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592627, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:28.139010 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.117227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:32987 (local address 127.5.227.190:46581)
0928 10:50:28.118011 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.118020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.119846 (+ 1826us) server_negotiation.cc:407] Connection header received
0928 10:50:28.121761 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.121765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.121835 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.121930 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.123833 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.124761 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.126873 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.127313 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.127436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.131934 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.131941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.131944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.131985 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.135302 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.136379 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.136391 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.136394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.136536 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.137029 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.137033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.137036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.137335 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.137506 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.138329 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.138580 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":427,"threads_started":1}
I20260928 10:50:28.140259 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.117979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46581 (local address 127.5.227.131:32987)
0928 10:50:28.118875 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:28.118934 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:28.121379 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:28.122263 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:28.122277 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:28.122914 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:28.123605 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.123627 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.124991 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.124995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.126632 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.126653 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.127888 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.127900 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.128178 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.128976 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:28.129003 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:28.131516 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:28.135481 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:28.135500 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:28.135516 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:28.136233 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:28.136663 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:28.136668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:28.136672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:28.136887 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:28.137477 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:28.137486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:28.137915 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 10:50:28.138222 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 10:50:28.138859 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":384,"threads_started":1}
I20260928 10:50:28.142436 7063 heartbeater.cc:347] Connected to a master server at 127.5.227.190:46581
I20260928 10:50:28.143033 7063 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:28.144261 7063 heartbeater.cc:511] Master 127.5.227.190:46581 requested a full tablet report, sending...
I20260928 10:50:28.146153 6606 ts_manager.cc:194] Registered new tserver with Master: d467b3da58f94938b38d0dd797999c98 (127.5.227.131:43681)
I20260928 10:50:28.147918 6606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:32987
I20260928 10:50:28.150462 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:50:28.177776 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.159360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.5.227.190:46581)
0928 10:50:28.159672 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.159682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.160115 (+ 433us) server_negotiation.cc:407] Connection header received
0928 10:50:28.160348 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.160354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.160451 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.160573 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.162893 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.163837 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.165764 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.166234 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.166423 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.169598 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.169604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.169607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.169652 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.173970 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.175159 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.175168 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.175172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.175361 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.176082 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.176090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.176094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.176502 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.176798 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.177195 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.177444 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260928 10:50:28.187417 6606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:50:28.196303 6606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:28.233757 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.214786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43303 (local address 127.0.0.1:41628)
0928 10:50:28.216419 (+ 1633us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:28.216491 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:28.216816 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:28.223506 (+ 6690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:28.223514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:28.223559 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:28.224807 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.224830 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.226792 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.226801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.229160 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.229179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.232605 (+ 3426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.232613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.232868 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.232928 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 10:50:28.233087 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 10:50:28.233328 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":937,"threads_started":1}
I20260928 10:50:28.233960 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.215048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.5.227.130:43303)
0928 10:50:28.222348 (+ 7300us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.222358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.222401 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:50:28.222816 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.222836 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.223184 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.223488 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:28.225059 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.226620 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.230437 (+ 3817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.232355 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.232525 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.232762 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.233217 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.233413 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7002,"thread_start_us":6891,"threads_started":1}
I20260928 10:50:28.236240 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.216012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.5.227.129:38415)
0928 10:50:28.221285 (+ 5273us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.221295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.221330 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:50:28.224037 (+ 2707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.224105 (+ 68us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.224428 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.226326 (+ 1898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:28.226906 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.229175 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.233587 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.235003 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.235187 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.235460 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.235690 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.235927 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4505,"thread_start_us":4100,"threads_started":1}
I20260928 10:50:28.236770 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.215440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:38415 (local address 127.0.0.1:58138)
0928 10:50:28.217723 (+ 2283us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:28.217758 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:28.217929 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:28.224887 (+ 6958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:28.224892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:28.224920 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:28.226201 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.226217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.229441 (+ 3224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.229834 (+ 393us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.233361 (+ 3527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.233375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.235239 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.235246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.235413 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.235455 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 10:50:28.235553 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 10:50:28.235707 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2130,"thread_start_us":1984,"threads_started":1}
I20260928 10:50:28.237421 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.222133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.5.227.131:43681)
0928 10:50:28.223183 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.223192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.224536 (+ 1344us) server_negotiation.cc:407] Connection header received
0928 10:50:28.224854 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.224863 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.225141 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.225352 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:28.226855 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.229190 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.233300 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.235479 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.235631 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.235940 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.236874 (+ 934us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.237053 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":720,"threads_started":1}
I20260928 10:50:28.237914 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.221754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:43681 (local address 127.0.0.1:50914)
0928 10:50:28.224207 (+ 2453us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:28.224229 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:28.224380 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:28.225324 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:28.225329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:28.225355 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:28.226201 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.226217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.229398 (+ 3181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.229403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.233017 (+ 3614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.233031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.236105 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.236112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.236315 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.236349 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:50:28.236449 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 10:50:28.236601 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2185,"mutex_wait_us":3,"thread_start_us":1729,"threads_started":1}
I20260928 10:50:28.239329 6864 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb581ed1eac4d509652bc307855b311 (DEFAULT_TABLE table=test-table [id=3b4d19f40e604f008c89450ec01e6905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:28.241179 6864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb581ed1eac4d509652bc307855b311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:28.240949 6730 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb581ed1eac4d509652bc307855b311 (DEFAULT_TABLE table=test-table [id=3b4d19f40e604f008c89450ec01e6905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:28.243199 6730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb581ed1eac4d509652bc307855b311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:28.242817 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb581ed1eac4d509652bc307855b311 (DEFAULT_TABLE table=test-table [id=3b4d19f40e604f008c89450ec01e6905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:28.245162 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb581ed1eac4d509652bc307855b311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:28.260231 7082 tablet_bootstrap.cc:492] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66: Bootstrap starting.
I20260928 10:50:28.264767 7082 tablet_bootstrap.cc:654] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:28.266441 7084 tablet_bootstrap.cc:492] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98: Bootstrap starting.
I20260928 10:50:28.266568 7082 log.cc:824] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:28.269361 7083 tablet_bootstrap.cc:492] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54: Bootstrap starting.
I20260928 10:50:28.273564 7084 tablet_bootstrap.cc:654] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:28.275009 7084 log.cc:824] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:28.275487 7083 tablet_bootstrap.cc:654] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:28.276413 7082 tablet_bootstrap.cc:492] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66: No bootstrap required, opened a new log
I20260928 10:50:28.276752 7082 ts_tablet_manager.cc:1397] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260928 10:50:28.277832 7083 log.cc:824] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:28.280666 7084 tablet_bootstrap.cc:492] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98: No bootstrap required, opened a new log
I20260928 10:50:28.281021 7084 ts_tablet_manager.cc:1397] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260928 10:50:28.282869 7083 tablet_bootstrap.cc:492] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54: No bootstrap required, opened a new log
I20260928 10:50:28.283425 7083 ts_tablet_manager.cc:1397] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260928 10:50:28.291365 7082 raft_consensus.cc:348] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.292037 7082 raft_consensus.cc:374] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:28.292284 7082 raft_consensus.cc:729] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e02cbd99e489d92d5725eab22cb66, State: Initialized, Role: FOLLOWER
I20260928 10:50:28.293105 7082 consensus_queue.cc:260] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.294922 7083 raft_consensus.cc:348] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.295096 7084 raft_consensus.cc:348] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.295533 7083 raft_consensus.cc:374] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:28.295655 7083 raft_consensus.cc:729] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c9ef76b7584606a49000ec67c69a54, State: Initialized, Role: FOLLOWER
I20260928 10:50:28.295836 7084 raft_consensus.cc:374] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:28.296007 7084 raft_consensus.cc:729] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d467b3da58f94938b38d0dd797999c98, State: Initialized, Role: FOLLOWER
I20260928 10:50:28.296067 7082 ts_tablet_manager.cc:1428] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66: Time spent starting tablet: real 0.019s user 0.013s sys 0.007s
I20260928 10:50:28.296602 7083 consensus_queue.cc:260] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.296867 7084 consensus_queue.cc:260] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.297886 6929 heartbeater.cc:503] Master 127.5.227.190:46581 was elected leader, sending a full tablet report...
I20260928 10:50:28.298492 7083 ts_tablet_manager.cc:1428] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54: Time spent starting tablet: real 0.015s user 0.006s sys 0.011s
I20260928 10:50:28.299611 7063 heartbeater.cc:503] Master 127.5.227.190:46581 was elected leader, sending a full tablet report...
I20260928 10:50:28.303200 7084 ts_tablet_manager.cc:1428] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98: Time spent starting tablet: real 0.022s user 0.018s sys 0.000s
I20260928 10:50:28.318025 7089 raft_consensus.cc:482] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:28.318380 7089 raft_consensus.cc:504] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.321743 7089 leader_election.cc:288] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 886e02cbd99e489d92d5725eab22cb66 (127.5.227.129:38415), d467b3da58f94938b38d0dd797999c98 (127.5.227.131:43681)
I20260928 10:50:28.330083 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.321233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:36773 (local address 127.5.227.131:43681)
0928 10:50:28.321470 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.321478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.322502 (+ 1024us) server_negotiation.cc:407] Connection header received
0928 10:50:28.322815 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.322822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.322896 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.323008 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:28.323693 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.324557 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.327577 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.328652 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.328921 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.329012 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.329664 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.329804 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:50:28.330278 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.321297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:45121 (local address 127.5.227.129:38415)
0928 10:50:28.321678 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.321685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.322173 (+ 488us) server_negotiation.cc:407] Connection header received
0928 10:50:28.322828 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.322835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.322929 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.323006 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:28.324000 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.324751 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.327455 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.328744 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.329102 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.329210 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.329654 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.329812 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260928 10:50:28.330302 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.321114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:43681 (local address 127.5.227.130:36773)
0928 10:50:28.322368 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:28.322393 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:28.322554 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:28.323029 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:28.323034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:28.323061 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:28.323460 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.323473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.324707 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.324716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.327192 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.327214 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.328952 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.328961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.329428 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.329463 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:50:28.329621 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 10:50:28.329777 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"mutex_wait_us":157,"thread_start_us":881,"threads_started":1}
I20260928 10:50:28.330518 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.321106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:38415 (local address 127.5.227.130:45121)
0928 10:50:28.322023 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:28.322044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:28.322419 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:28.323235 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:28.323240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:28.323268 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:28.323774 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.323789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.324954 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.324966 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.327216 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:28.327232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.329177 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.329187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:28.329444 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.329476 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:50:28.329604 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 10:50:28.329812 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":175,"thread_start_us":502,"threads_started":1}
I20260928 10:50:28.331931 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb581ed1eac4d509652bc307855b311" candidate_uuid: "58c9ef76b7584606a49000ec67c69a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d467b3da58f94938b38d0dd797999c98" is_pre_election: true
I20260928 10:50:28.333276 7018 raft_consensus.cc:2454] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c9ef76b7584606a49000ec67c69a54 in term 0.
I20260928 10:50:28.333191 6750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb581ed1eac4d509652bc307855b311" candidate_uuid: "58c9ef76b7584606a49000ec67c69a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e02cbd99e489d92d5725eab22cb66" is_pre_election: true
I20260928 10:50:28.333801 6750 raft_consensus.cc:2454] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c9ef76b7584606a49000ec67c69a54 in term 0.
I20260928 10:50:28.334450 6819 leader_election.cc:302] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c9ef76b7584606a49000ec67c69a54, d467b3da58f94938b38d0dd797999c98; no voters:
I20260928 10:50:28.335301 7089 raft_consensus.cc:2781] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:28.335480 7089 raft_consensus.cc:482] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:28.335554 7089 raft_consensus.cc:3037] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:28.338982 7089 raft_consensus.cc:504] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.339892 7089 leader_election.cc:288] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [CANDIDATE]: Term 1 election: Requested vote from peers 886e02cbd99e489d92d5725eab22cb66 (127.5.227.129:38415), d467b3da58f94938b38d0dd797999c98 (127.5.227.131:43681)
I20260928 10:50:28.340595 6750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb581ed1eac4d509652bc307855b311" candidate_uuid: "58c9ef76b7584606a49000ec67c69a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e02cbd99e489d92d5725eab22cb66"
I20260928 10:50:28.340862 6750 raft_consensus.cc:3037] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:28.340652 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb581ed1eac4d509652bc307855b311" candidate_uuid: "58c9ef76b7584606a49000ec67c69a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d467b3da58f94938b38d0dd797999c98"
I20260928 10:50:28.341354 7018 raft_consensus.cc:3037] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98 [term 0 FOLLOWER]: Advancing to term 1
W20260928 10:50:28.343394 6796 tablet.cc:2369] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:50:28.344854 6750 raft_consensus.cc:2454] T 4fb581ed1eac4d509652bc307855b311 P 886e02cbd99e489d92d5725eab22cb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c9ef76b7584606a49000ec67c69a54 in term 1.
I20260928 10:50:28.345571 6818 leader_election.cc:302] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c9ef76b7584606a49000ec67c69a54, 886e02cbd99e489d92d5725eab22cb66; no voters:
I20260928 10:50:28.345866 7018 raft_consensus.cc:2454] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c9ef76b7584606a49000ec67c69a54 in term 1.
I20260928 10:50:28.346239 7089 raft_consensus.cc:2781] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:28.347378 7089 raft_consensus.cc:686] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [term 1 LEADER]: Becoming Leader. State: Replica: 58c9ef76b7584606a49000ec67c69a54, State: Running, Role: LEADER
I20260928 10:50:28.348384 7089 consensus_queue.cc:237] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } }
I20260928 10:50:28.354522 6606 catalog_manager.cc:5495] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58c9ef76b7584606a49000ec67c69a54 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "58c9ef76b7584606a49000ec67c69a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c9ef76b7584606a49000ec67c69a54" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e02cbd99e489d92d5725eab22cb66" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d467b3da58f94938b38d0dd797999c98" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 43681 } health_report { overall_health: UNKNOWN } } }
W20260928 10:50:28.366039 7064 tablet.cc:2369] T 4fb581ed1eac4d509652bc307855b311 P d467b3da58f94938b38d0dd797999c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:28.405539 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.383142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.5.227.129:38415)
0928 10:50:28.383437 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.383444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.384047 (+ 603us) server_negotiation.cc:407] Connection header received
0928 10:50:28.384593 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.384600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.384834 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.385029 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.388448 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.389292 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.390778 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.391230 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.391427 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.396864 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.396883 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.396901 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.396947 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.401235 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.402284 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.402293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.402296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.402464 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.403512 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.403517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.403520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.404067 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.404492 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.405001 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.405215 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 10:50:28.454028 6930 tablet.cc:2369] T 4fb581ed1eac4d509652bc307855b311 P 58c9ef76b7584606a49000ec67c69a54: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:28.469743 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.446976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.5.227.129:38415)
0928 10:50:28.447227 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.447237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.447467 (+ 230us) server_negotiation.cc:407] Connection header received
0928 10:50:28.447598 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.447603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.447944 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.448043 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.449944 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.450891 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.452900 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.453590 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.453832 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.459284 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.459291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.459297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.459350 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.464557 (+ 5207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.465776 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.465783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.465787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.465967 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.466539 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.466545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.466548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.467282 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.467819 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.468592 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.469291 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:28.529377 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.508141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.5.227.129:38415)
0928 10:50:28.508399 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.508406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.508706 (+ 300us) server_negotiation.cc:407] Connection header received
0928 10:50:28.508806 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.508813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.508897 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.509052 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.510883 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.511993 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.514373 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.515140 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.515272 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.520952 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.520958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.520961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.521005 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.524804 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.525886 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.525898 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.525901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.526072 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.526778 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.526807 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.526810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.527257 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.527592 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.528498 (+ 906us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.528835 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260928 10:50:28.531525 6710 tablet_service.cc:2908] Rejecting scan request for tablet 4fb581ed1eac4d509652bc307855b311: Uninitialized: clean time has not yet been initialized
I20260928 10:50:28.536120 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6667
I20260928 10:50:28.554929 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.537577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.5.227.129:38415)
0928 10:50:28.537858 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.537865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.538131 (+ 266us) server_negotiation.cc:407] Connection header received
0928 10:50:28.538249 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.538255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.538347 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.538439 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.540164 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.541003 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.542620 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.543242 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.543386 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.547008 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.547023 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.547027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.547076 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.550629 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.551907 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.551911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.551913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.552054 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.553028 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.553032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.553034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.553385 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.553766 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.554165 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.554682 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 10:50:28.555892 6790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:28.659816 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6667
I20260928 10:50:28.681459 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6801
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:50:28.701813 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.682831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.5.227.130:43303)
0928 10:50:28.683041 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.683045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.683064 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:50:28.683120 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.683124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.683192 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.683286 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.685135 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.686038 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.687954 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.688439 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.688573 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.693324 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.693343 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.693361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.693406 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.697841 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.699172 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.699181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.699184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.699395 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.700072 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.700077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.700079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.700619 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.700890 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.701318 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.701549 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:50:28.702893 6924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:28.808894 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6801
I20260928 10:50:28.828552 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6935
Sep 28 10:50:28 dist-test-slave-8w8g krb5kdc[6561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:50:28.848356 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:28.829600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.5.227.131:43681)
0928 10:50:28.829914 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:28.829922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:28.829952 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:50:28.830316 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:28.830329 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:28.830416 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:28.830672 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:28.832372 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.833224 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.834618 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:28.835024 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:28.835183 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:28.840532 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:28.840557 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:28.840579 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:28.840828 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:28.845003 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.845932 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.845941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.845944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.846127 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:28.846626 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:28.846631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:28.846633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:28.847020 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:28.847336 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:28.847675 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 10:50:28.847958 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260928 10:50:28.849128 7058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:28.949431 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6935
I20260928 10:50:28.968145 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 6574
I20260928 10:50:28.969463 6636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:29.082935 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 6574
2026-09-28T10:50:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4884 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:29 dist-test-slave-8w8g krb5kdc[7112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:29 dist-test-slave-8w8g krb5kdc[7112](info): set up 2 sockets
Sep 28 10:50:29 dist-test-slave-8w8g krb5kdc[7112](info): commencing operation
krb5kdc: starting...
W20260928 10:50:30.332962 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:30 dist-test-slave-8w8g krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:30.587265 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39843
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:30.994848 7125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:30.995318 7125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:30.995437 7125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:31.003587 7125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:31.003762 7125 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:50:31.003806 7125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:31.004163 7125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:31.004398 7125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:31.004523 7125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:31.015581 7125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39843
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:31.017164 7125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:31.018664 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:31.031149 7130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:31.031122 7131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:31.031352 7133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:31.031898 7125 server_base.cc:1034] running on GCE node
I20260928 10:50:31.032745 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:31.034241 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:31.035496 7125 hybrid_clock.cc:648] HybridClock initialized: now 1790592631035428 us; error 50 us; skew 500 ppm
I20260928 10:50:31.035908 7125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:31 dist-test-slave-8w8g krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592631, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:31.044878 7125 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:31.046609 7125 webserver.cc:460] Webserver started at http://127.5.227.190:46539/ using document root <none> and password file <none>
I20260928 10:50:31.047413 7125 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:31.047554 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:31.047979 7125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:31.051218 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "add823eb42e948249afbc392ba2b7973"
format_stamp: "Formatted at 2026-09-28 10:50:31 on dist-test-slave-8w8g"
server_key: "73cc56097207e24d1a9781a25ce45087"
server_key_iv: "6c3ed3fcaea88747a618cc2a71243df9"
server_key_version: "encryptionkey@0"
I20260928 10:50:31.052145 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "add823eb42e948249afbc392ba2b7973"
format_stamp: "Formatted at 2026-09-28 10:50:31 on dist-test-slave-8w8g"
server_key: "73cc56097207e24d1a9781a25ce45087"
server_key_iv: "6c3ed3fcaea88747a618cc2a71243df9"
server_key_version: "encryptionkey@0"
I20260928 10:50:31.057695 7125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 10:50:31.061504 7140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:31.063067 7125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:50:31.063335 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "add823eb42e948249afbc392ba2b7973"
format_stamp: "Formatted at 2026-09-28 10:50:31 on dist-test-slave-8w8g"
server_key: "73cc56097207e24d1a9781a25ce45087"
server_key_iv: "6c3ed3fcaea88747a618cc2a71243df9"
server_key_version: "encryptionkey@0"
I20260928 10:50:31.063593 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:31.087355 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:31.092900 7125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:31.093252 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:31.120537 7125 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:39843
I20260928 10:50:31.120550 7201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:39843 every 8 connection(s)
I20260928 10:50:31.122560 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:31.128206 7202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:31.132241 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 7125
I20260928 10:50:31.132527 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:31.133399 6030 external_mini_cluster.cc:1428] Setting key 59e67c23582dc86730bdab8876ce7aad
I20260928 10:50:31.143931 7202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973: Bootstrap starting.
I20260928 10:50:31.148694 7202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:31.150235 7202 log.cc:824] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:31.154578 7202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973: No bootstrap required, opened a new log
Sep 28 10:50:31 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:31.168659 7202 raft_consensus.cc:348] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add823eb42e948249afbc392ba2b7973" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39843 } }
I20260928 10:50:31.169194 7202 raft_consensus.cc:374] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:31.169231 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:31.136335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.5.227.190:39843)
0928 10:50:31.137387 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:31.137408 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:31.137474 (+ 66us) server_negotiation.cc:407] Connection header received
0928 10:50:31.141055 (+ 3581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:31.141104 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:31.142238 (+ 1134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:31.142971 (+ 733us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:31.144878 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.147248 (+ 2370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.149208 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.150442 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.150662 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:31.156553 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:31.156573 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:31.156598 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:31.156664 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:31.162226 (+ 5562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:31.163465 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:31.163478 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:31.163488 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:31.163806 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:31.164462 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:31.164468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:31.164472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:31.165363 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:31.165934 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:31.166877 (+ 943us) server_negotiation.cc:299] Negotiation successful
0928 10:50:31.167487 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":417,"threads_started":1}
I20260928 10:50:31.169399 7202 raft_consensus.cc:729] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add823eb42e948249afbc392ba2b7973, State: Initialized, Role: FOLLOWER
I20260928 10:50:31.170083 7202 consensus_queue.cc:260] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add823eb42e948249afbc392ba2b7973" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39843 } }
I20260928 10:50:31.170394 7202 raft_consensus.cc:388] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:31.170564 7202 raft_consensus.cc:482] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:31.170784 7202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:31.174615 7202 raft_consensus.cc:504] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add823eb42e948249afbc392ba2b7973" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39843 } }
I20260928 10:50:31.175311 7202 leader_election.cc:302] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: add823eb42e948249afbc392ba2b7973; no voters:
I20260928 10:50:31.176169 7202 leader_election.cc:288] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:31.176352 7207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:31.177847 7207 raft_consensus.cc:686] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [term 1 LEADER]: Becoming Leader. State: Replica: add823eb42e948249afbc392ba2b7973, State: Running, Role: LEADER
I20260928 10:50:31.178580 7207 consensus_queue.cc:237] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add823eb42e948249afbc392ba2b7973" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39843 } }
I20260928 10:50:31.178946 7202 sys_catalog.cc:564] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:31.181610 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader add823eb42e948249afbc392ba2b7973. Latest consensus state: current_term: 1 leader_uuid: "add823eb42e948249afbc392ba2b7973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add823eb42e948249afbc392ba2b7973" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39843 } } }
I20260928 10:50:31.182262 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:31.182121 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "add823eb42e948249afbc392ba2b7973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add823eb42e948249afbc392ba2b7973" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39843 } } }
I20260928 10:50:31.182708 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:31.183943 7216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:31.192214 7216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:31.204442 7216 catalog_manager.cc:1347] Generated new cluster ID: 2040a14239c14a08a5f0d029ec7c78ea
I20260928 10:50:31.204689 7216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:31.223171 7216 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:31.225978 7216 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:31.234874 7216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973: Generated new TSK 0
I20260928 10:50:31.236382 7216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:50:31.311344 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39843
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:31.724921 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:31.725430 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:31.725628 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:31.736325 7228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:31.736606 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:31.736832 7228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:50:31.736994 7228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:50:31.749725 7228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:39843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:31.751510 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:31.754081 7228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:31.773324 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:31.775141 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:31.775220 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:31.776003 7228 server_base.cc:1034] running on GCE node
I20260928 10:50:31.776942 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:31.778479 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:31.782126 7228 hybrid_clock.cc:648] HybridClock initialized: now 1790592631782147 us; error 190 us; skew 500 ppm
I20260928 10:50:31.782579 7228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:31 dist-test-slave-8w8g krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592631, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:31.786656 7228 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:50:31.788471 7228 webserver.cc:460] Webserver started at http://127.5.227.129:42069/ using document root <none> and password file <none>
I20260928 10:50:31.789260 7228 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:31.789371 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:31.789690 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:31.794471 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "f8ec49710adf48f39982a06309948c60"
format_stamp: "Formatted at 2026-09-28 10:50:31 on dist-test-slave-8w8g"
server_key: "9d4f77f8444ab0045861fb17f5f43a1f"
server_key_iv: "f5263116d1644ad5cb1a636628ca3a68"
server_key_version: "encryptionkey@0"
I20260928 10:50:31.796065 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "f8ec49710adf48f39982a06309948c60"
format_stamp: "Formatted at 2026-09-28 10:50:31 on dist-test-slave-8w8g"
server_key: "9d4f77f8444ab0045861fb17f5f43a1f"
server_key_iv: "f5263116d1644ad5cb1a636628ca3a68"
server_key_version: "encryptionkey@0"
I20260928 10:50:31.803212 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260928 10:50:31.807713 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:31.810077 7228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:50:31.810316 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "f8ec49710adf48f39982a06309948c60"
format_stamp: "Formatted at 2026-09-28 10:50:31 on dist-test-slave-8w8g"
server_key: "9d4f77f8444ab0045861fb17f5f43a1f"
server_key_iv: "f5263116d1644ad5cb1a636628ca3a68"
server_key_version: "encryptionkey@0"
I20260928 10:50:31.810809 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:31.834830 7228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:31.840037 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:31.840358 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:31.843412 7228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:31.843556 7228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:31.844009 7228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:31.844096 7228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:50:31 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592631, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:31.881610 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:31.845015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:35105 (local address 127.5.227.190:39843)
0928 10:50:31.847470 (+ 2455us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:31.847480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:31.849016 (+ 1536us) server_negotiation.cc:407] Connection header received
0928 10:50:31.852427 (+ 3411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:31.852434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:31.852547 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:31.852811 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:31.856641 (+ 3830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.857654 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.861466 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.861926 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.862067 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:31.871122 (+ 9055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:31.871134 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:31.871141 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:31.871215 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:31.876306 (+ 5091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:31.877733 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:31.877739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:31.877742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:31.877895 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:31.878451 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:31.878456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:31.878458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:31.878791 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:31.878969 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:31.880926 (+ 1957us) server_negotiation.cc:299] Negotiation successful
0928 10:50:31.881226 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2308,"thread_start_us":2187,"threads_started":1}
I20260928 10:50:31.883230 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:31.846913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39843 (local address 127.5.227.129:35105)
0928 10:50:31.848474 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:31.848764 (+ 290us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:31.851974 (+ 3210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:31.853350 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:31.853368 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:31.854849 (+ 1481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:31.856294 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:31.856342 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.857855 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.857862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:31.861174 (+ 3312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:31.861209 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.862284 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.862296 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:31.862786 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:31.864337 (+ 1551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:31.864381 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:31.870650 (+ 6269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:31.876621 (+ 5971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:31.876637 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:31.876661 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:31.877501 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:31.878034 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:31.878039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:31.878042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:31.878300 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:31.878940 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:31.878949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:31.880065 (+ 1116us) client_negotiation.cc:769] Sending connection context
0928 10:50:31.880793 (+ 728us) client_negotiation.cc:240] Negotiation successful
0928 10:50:31.881480 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"spinlock_wait_cycles":20864,"thread_start_us":955,"threads_started":1}
I20260928 10:50:31.912424 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:31.892912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39843 (local address 127.5.227.129:49435)
0928 10:50:31.893812 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:31.893899 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:31.894342 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:31.894730 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:31.894735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:31.895267 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:31.895568 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:31.895593 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.896803 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.896812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:31.899565 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:31.899595 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.901057 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.901065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:31.901235 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:31.901944 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:31.901963 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:31.902775 (+ 812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:31.907085 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:31.907093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:31.907103 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:31.908033 (+ 930us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:31.908807 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:31.908815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:31.908822 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:31.909070 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:31.910324 (+ 1254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:31.910329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:31.910724 (+ 395us) client_negotiation.cc:769] Sending connection context
0928 10:50:31.911008 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 10:50:31.911896 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":378,"threads_started":1}
I20260928 10:50:31.912575 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:31.892656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:49435 (local address 127.5.227.190:39843)
0928 10:50:31.892911 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:31.892919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:31.894264 (+ 1345us) server_negotiation.cc:407] Connection header received
0928 10:50:31.894506 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:31.894510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:31.894578 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:31.894698 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:31.895794 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.896594 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.899888 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:31.900827 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:31.900984 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:31.903019 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:31.903025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:31.903028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:31.903062 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:31.906822 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:31.908242 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:31.908251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:31.908256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:31.908454 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:31.909338 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:31.909344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:31.909351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:31.910129 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:31.910347 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:31.911948 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0928 10:50:31.912228 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:50:31.917819 7228 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:37963
I20260928 10:50:31.917829 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:37963 every 8 connection(s)
I20260928 10:50:31.919384 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:50:31.922106 7364 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39843
I20260928 10:50:31.922376 7364 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:31.923061 7364 heartbeater.cc:511] Master 127.5.227.190:39843 requested a full tablet report, sending...
I20260928 10:50:31.925388 7157 ts_manager.cc:194] Registered new tserver with Master: f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963)
I20260928 10:50:31.927034 7157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:35105
I20260928 10:50:31.929484 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 7228
I20260928 10:50:31.929829 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:50:31.930528 6030 external_mini_cluster.cc:1428] Setting key b7655dd26e609a2e724bd13ddfde1035
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:50:31.994885 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39843
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:32.432904 7368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:32.433525 7368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:32.433794 7368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:32.442270 7368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:32.442572 7368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:32.442950 7368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:50:32.443126 7368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:50:32.455646 7368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:39843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:32.457285 7368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:32.458943 7368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 10:50:32.476850 7368 server_base.cc:1034] running on GCE node
W20260928 10:50:32.476899 7374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:32.476850 7376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:32.476837 7373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:32.478072 7368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:32.479802 7368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:32.481189 7368 hybrid_clock.cc:648] HybridClock initialized: now 1790592632481086 us; error 70 us; skew 500 ppm
I20260928 10:50:32.481683 7368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:32 dist-test-slave-8w8g krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592632, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:32.486404 7368 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:50:32.488363 7368 webserver.cc:460] Webserver started at http://127.5.227.130:35337/ using document root <none> and password file <none>
I20260928 10:50:32.489320 7368 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:32.489434 7368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:32.489797 7368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:32.493427 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "1be4453736ac41bf96f202ff66e9b31e"
format_stamp: "Formatted at 2026-09-28 10:50:32 on dist-test-slave-8w8g"
server_key: "d72a3b2fc689f46e5ae08dc4923d6a76"
server_key_iv: "0651a628c56a6840c54ac2141a2d6323"
server_key_version: "encryptionkey@0"
I20260928 10:50:32.494489 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "1be4453736ac41bf96f202ff66e9b31e"
format_stamp: "Formatted at 2026-09-28 10:50:32 on dist-test-slave-8w8g"
server_key: "d72a3b2fc689f46e5ae08dc4923d6a76"
server_key_iv: "0651a628c56a6840c54ac2141a2d6323"
server_key_version: "encryptionkey@0"
I20260928 10:50:32.502720 7368 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260928 10:50:32.508190 7383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:32.510684 7368 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 10:50:32.510943 7368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "1be4453736ac41bf96f202ff66e9b31e"
format_stamp: "Formatted at 2026-09-28 10:50:32 on dist-test-slave-8w8g"
server_key: "d72a3b2fc689f46e5ae08dc4923d6a76"
server_key_iv: "0651a628c56a6840c54ac2141a2d6323"
server_key_version: "encryptionkey@0"
I20260928 10:50:32.511225 7368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:32.534605 7368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:32.539925 7368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:32.540319 7368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:32.544123 7368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:32.544271 7368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:32.544423 7368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:32.544530 7368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:50:32 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592632, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:32.573865 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:32.544239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:45095 (local address 127.5.227.190:39843)
0928 10:50:32.544923 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:32.544934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:32.546374 (+ 1440us) server_negotiation.cc:407] Connection header received
0928 10:50:32.549806 (+ 3432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:32.549813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:32.549916 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:32.550060 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:32.553100 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.554272 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.557347 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.557802 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.557972 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:32.565390 (+ 7418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:32.565398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:32.565405 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:32.565450 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:32.569659 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:32.570895 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:32.570903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:32.570909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:32.571104 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:32.571792 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:32.571797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:32.571801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:32.572124 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:32.572325 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:32.573218 (+ 893us) server_negotiation.cc:299] Negotiation successful
0928 10:50:32.573472 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":401,"threads_started":1}
I20260928 10:50:32.575042 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:32.545099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39843 (local address 127.5.227.130:45095)
0928 10:50:32.546128 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:32.546200 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:32.549312 (+ 3112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:32.550600 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:32.550624 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:32.551503 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:32.552646 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:32.552704 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.554508 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.554517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:32.557087 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:32.557109 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.558092 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.558107 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:32.558357 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:32.559265 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:32.559312 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:32.564878 (+ 5566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:32.569871 (+ 4993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:32.569883 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:32.569907 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:32.570695 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:32.571256 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:32.571261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:32.571264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:32.571571 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:32.572295 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:32.572304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:32.572783 (+ 479us) client_negotiation.cc:769] Sending connection context
0928 10:50:32.573075 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 10:50:32.573704 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":359,"threads_started":1}
I20260928 10:50:32.601392 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:32.582085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:46389 (local address 127.5.227.190:39843)
0928 10:50:32.582937 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:32.582946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:32.583533 (+ 587us) server_negotiation.cc:407] Connection header received
0928 10:50:32.584136 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:32.584140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:32.584221 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:32.584334 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:32.585390 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.586208 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.588022 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.588554 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.588711 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:32.593380 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:32.593386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:32.593390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:32.593429 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:32.596576 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:32.597821 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:32.597830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:32.597836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:32.598034 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:32.598726 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:32.598735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:32.598742 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:32.599387 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:32.599692 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:32.600690 (+ 998us) server_negotiation.cc:299] Negotiation successful
0928 10:50:32.600970 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344}
I20260928 10:50:32.601480 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:32.581972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39843 (local address 127.5.227.130:46389)
0928 10:50:32.583316 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:32.583386 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:32.583918 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:32.584364 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:32.584368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:32.584896 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:32.585213 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:32.585226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.586357 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.586363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:32.587672 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:32.587686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:32.588862 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:32.588873 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:32.589156 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:32.590710 (+ 1554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:32.590792 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:32.593024 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:32.596757 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:32.596763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:32.596768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:32.597481 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:32.598232 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:32.598239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:32.598246 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:32.598472 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:32.599898 (+ 1426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:32.599903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:32.600318 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 10:50:32.600537 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 10:50:32.601122 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":459,"threads_started":1}
I20260928 10:50:32.612764 7368 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:43897
I20260928 10:50:32.612880 7503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:43897 every 8 connection(s)
I20260928 10:50:32.614203 7368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:50:32.617820 7504 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39843
I20260928 10:50:32.618136 7504 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:32.618891 7504 heartbeater.cc:511] Master 127.5.227.190:39843 requested a full tablet report, sending...
I20260928 10:50:32.620931 7157 ts_manager.cc:194] Registered new tserver with Master: 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130:43897)
I20260928 10:50:32.621869 7157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:45095
I20260928 10:50:32.623193 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 7368
I20260928 10:50:32.623405 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:50:32.623924 6030 external_mini_cluster.cc:1428] Setting key fd001105eca3de4470caa7eeb817405c
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:50:32.691385 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39843
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:50:32.931357 7364 heartbeater.cc:503] Master 127.5.227.190:39843 was elected leader, sending a full tablet report...
W20260928 10:50:33.078222 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:33.078614 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:33.078807 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:33.087041 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:33.087176 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:33.087266 7508 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:50:33.087355 7508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:50:33.098505 7508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:39843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:33.100792 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:33.102631 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:33.119807 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:33.119765 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:33.119683 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:33.127170 7508 server_base.cc:1034] running on GCE node
I20260928 10:50:33.128376 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:33.129756 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:33.131345 7508 hybrid_clock.cc:648] HybridClock initialized: now 1790592633131225 us; error 77 us; skew 500 ppm
I20260928 10:50:33.131879 7508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:33 dist-test-slave-8w8g krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592633, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:33.136926 7508 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:50:33.139854 7508 webserver.cc:460] Webserver started at http://127.5.227.131:46519/ using document root <none> and password file <none>
I20260928 10:50:33.140913 7508 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:33.141098 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:33.141476 7508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:33.146023 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "34ee37630bf1401a8f10034eb87e0646"
format_stamp: "Formatted at 2026-09-28 10:50:33 on dist-test-slave-8w8g"
server_key: "cc98ac0eb4020216d6ab52f12205c73b"
server_key_iv: "f8fb0ac5bcccf08b76b545df314ba9b7"
server_key_version: "encryptionkey@0"
I20260928 10:50:33.147217 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "34ee37630bf1401a8f10034eb87e0646"
format_stamp: "Formatted at 2026-09-28 10:50:33 on dist-test-slave-8w8g"
server_key: "cc98ac0eb4020216d6ab52f12205c73b"
server_key_iv: "f8fb0ac5bcccf08b76b545df314ba9b7"
server_key_version: "encryptionkey@0"
I20260928 10:50:33.154721 7508 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 10:50:33.159873 7523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:33.161923 7508 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:50:33.162352 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "34ee37630bf1401a8f10034eb87e0646"
format_stamp: "Formatted at 2026-09-28 10:50:33 on dist-test-slave-8w8g"
server_key: "cc98ac0eb4020216d6ab52f12205c73b"
server_key_iv: "f8fb0ac5bcccf08b76b545df314ba9b7"
server_key_version: "encryptionkey@0"
I20260928 10:50:33.162817 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:33.185772 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:33.191479 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:33.191859 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:33.196746 7508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:33.197011 7508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.001s
I20260928 10:50:33.197222 7508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:33.197320 7508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:50:33 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592633, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:33.229794 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.196949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39843 (local address 127.5.227.131:35331)
0928 10:50:33.198051 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.198110 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.201320 (+ 3210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.202526 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.202541 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.203793 (+ 1252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:33.205110 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.205153 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.206555 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.206564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.209980 (+ 3416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.210016 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.211088 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.211105 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.211614 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.212749 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:33.212799 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:33.217861 (+ 5062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:33.222484 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:33.222496 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:33.222522 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:33.223098 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:33.223563 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:33.223568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:33.223685 (+ 117us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:33.224090 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:33.225005 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:33.225019 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:33.225813 (+ 794us) client_negotiation.cc:769] Sending connection context
0928 10:50:33.226279 (+ 466us) client_negotiation.cc:240] Negotiation successful
0928 10:50:33.227395 (+ 1116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":423,"threads_started":1}
I20260928 10:50:33.231925 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.196296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:35331 (local address 127.5.227.190:39843)
0928 10:50:33.197068 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.197120 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.198278 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 10:50:33.201887 (+ 3609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.201895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.202068 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.202208 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:33.205471 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.206312 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.210284 (+ 3972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.210742 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.210873 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.218317 (+ 7444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:33.218328 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:33.218334 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:33.218383 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:33.222275 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:33.223279 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:33.223283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:33.223286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:33.223429 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:33.224322 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:33.224330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:33.224338 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:33.224782 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:33.230568 (+ 5786us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.230875 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.231272 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":514,"threads_started":1}
I20260928 10:50:33.255368 7508 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:41539
I20260928 10:50:33.255436 7643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:41539 every 8 connection(s)
I20260928 10:50:33.257124 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:50:33.260274 7644 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39843
I20260928 10:50:33.260694 7644 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:33.261559 7644 heartbeater.cc:511] Master 127.5.227.190:39843 requested a full tablet report, sending...
I20260928 10:50:33.263420 7157 ts_manager.cc:194] Registered new tserver with Master: 34ee37630bf1401a8f10034eb87e0646 (127.5.227.131:41539)
I20260928 10:50:33.263484 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.243517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39843 (local address 127.5.227.131:33417)
0928 10:50:33.245126 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.245150 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.245372 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.246203 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.246212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.247119 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:33.248123 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.248151 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.250133 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.250143 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.252461 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.252483 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.253446 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.253455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.253690 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.254765 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:33.254801 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:33.256123 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:33.260762 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:33.260768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:33.260772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:33.261263 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:33.261707 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:33.261713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:33.261716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:33.261853 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:33.262435 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:33.262439 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:33.262744 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 10:50:33.262905 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 10:50:33.263173 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"thread_start_us":1266,"threads_started":1}
I20260928 10:50:33.264003 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.243443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:33417 (local address 127.5.227.190:39843)
0928 10:50:33.243643 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.243648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.245291 (+ 1643us) server_negotiation.cc:407] Connection header received
0928 10:50:33.245536 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.245544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.245657 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.245779 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:33.248499 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.249817 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.252757 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.253209 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.253369 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.256429 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:33.256437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:33.256442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:33.256490 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:33.260616 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:33.261420 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:33.261426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:33.261431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:33.261577 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:33.261980 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:33.261985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:33.261989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:33.262307 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:33.262477 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.263092 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.263364 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 10:50:33.264863 7157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:35331
I20260928 10:50:33.266324 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 7508
I20260928 10:50:33.266533 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:50:33.267132 6030 external_mini_cluster.cc:1428] Setting key e6b286249e28283cfc8178db082fed11
I20260928 10:50:33.271034 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:50:33 dist-test-slave-8w8g krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:50:33 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:33.323976 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.307006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.5.227.190:39843)
0928 10:50:33.307233 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.307239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.307401 (+ 162us) server_negotiation.cc:407] Connection header received
0928 10:50:33.307673 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.307677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.307807 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.307919 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:33.309393 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.310144 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.311708 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.312268 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.312398 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.317394 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:33.317400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:33.317404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:33.317596 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:33.320615 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:33.321684 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:33.321689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:33.321692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:33.321855 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:33.322377 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:33.322382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:33.322385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:33.322725 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:33.322943 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.323408 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.323614 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 10:50:33.329950 7156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:50:33.339085 7156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:33.374645 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.354070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43897 (local address 127.0.0.1:42532)
0928 10:50:33.356408 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.356458 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.356660 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.366111 (+ 9451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.366119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.366150 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:33.366929 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.366951 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.369115 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.369121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.371451 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.371469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.373417 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.373423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.373648 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.373725 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 10:50:33.373862 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 10:50:33.374077 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2028,"mutex_wait_us":32,"thread_start_us":10887,"threads_started":1}
I20260928 10:50:33.374714 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.355021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.5.227.129:37963)
0928 10:50:33.364908 (+ 9887us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.364919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.365364 (+ 445us) server_negotiation.cc:407] Connection header received
0928 10:50:33.365627 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.365635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.365888 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.366215 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:33.367964 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.369602 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.372068 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.373618 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.373760 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.374013 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.374225 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.374411 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9708,"thread_start_us":9571,"threads_started":1}
I20260928 10:50:33.374754 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.354402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37963 (local address 127.0.0.1:52916)
0928 10:50:33.363680 (+ 9278us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.363713 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.365443 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.366346 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.366351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.366371 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:33.367659 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.367672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.369768 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.369773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.371877 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.371893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.373849 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.373857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.374019 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.374050 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:50:33.374127 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 10:50:33.374275 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9058,"mutex_wait_us":160,"thread_start_us":2499,"threads_started":1}
I20260928 10:50:33.375864 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.354161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.5.227.130:43897)
0928 10:50:33.365329 (+ 11168us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.365337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.365370 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:50:33.365498 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.365508 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.365840 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.366089 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:33.367172 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.368968 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.371651 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.373189 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.375096 (+ 1907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.375313 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.375378 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.375568 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11020,"thread_start_us":10907,"threads_started":1}
I20260928 10:50:33.375965 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.355995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:41539 (local address 127.0.0.1:56220)
0928 10:50:33.365335 (+ 9340us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.365356 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.365498 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.366491 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.366496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.366541 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:33.368108 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.368122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.370552 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.370558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.372803 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.372819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.375086 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.375095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.375320 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.375359 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:50:33.375449 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 10:50:33.375594 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9173,"mutex_wait_us":19,"thread_start_us":7229,"threads_started":1}
I20260928 10:50:33.376263 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.356599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.5.227.131:41539)
0928 10:50:33.364600 (+ 8001us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.364616 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.365453 (+ 837us) server_negotiation.cc:407] Connection header received
0928 10:50:33.365665 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.365672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.365901 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.366216 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:33.368787 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.370368 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.373034 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.374556 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.374689 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.374938 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.375542 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.375714 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7734,"thread_start_us":7500,"threads_started":1}
I20260928 10:50:33.379680 7294 tablet_service.cc:1467] Processing CreateTablet for tablet 32283fce68dd4bcaa6b7d92d4cd4452b (DEFAULT_TABLE table=test-table [id=eabaa59151514b0e97bfa7d3e078ba71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:33.380050 7574 tablet_service.cc:1467] Processing CreateTablet for tablet 32283fce68dd4bcaa6b7d92d4cd4452b (DEFAULT_TABLE table=test-table [id=eabaa59151514b0e97bfa7d3e078ba71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:33.379772 7434 tablet_service.cc:1467] Processing CreateTablet for tablet 32283fce68dd4bcaa6b7d92d4cd4452b (DEFAULT_TABLE table=test-table [id=eabaa59151514b0e97bfa7d3e078ba71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:33.381646 7294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32283fce68dd4bcaa6b7d92d4cd4452b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:33.381865 7434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32283fce68dd4bcaa6b7d92d4cd4452b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:33.381863 7574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32283fce68dd4bcaa6b7d92d4cd4452b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:33.398262 7663 tablet_bootstrap.cc:492] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Bootstrap starting.
I20260928 10:50:33.402518 7663 tablet_bootstrap.cc:654] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:33.404207 7664 tablet_bootstrap.cc:492] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Bootstrap starting.
I20260928 10:50:33.404515 7663 log.cc:824] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:33.404855 7665 tablet_bootstrap.cc:492] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Bootstrap starting.
I20260928 10:50:33.408553 7664 tablet_bootstrap.cc:654] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:33.409457 7663 tablet_bootstrap.cc:492] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: No bootstrap required, opened a new log
I20260928 10:50:33.409777 7663 ts_tablet_manager.cc:1397] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260928 10:50:33.409785 7664 log.cc:824] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:33.410297 7665 tablet_bootstrap.cc:654] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:33.412252 7665 log.cc:824] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:33.412616 7664 tablet_bootstrap.cc:492] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: No bootstrap required, opened a new log
I20260928 10:50:33.412931 7664 ts_tablet_manager.cc:1397] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 10:50:33.416092 7665 tablet_bootstrap.cc:492] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: No bootstrap required, opened a new log
I20260928 10:50:33.416513 7665 ts_tablet_manager.cc:1397] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260928 10:50:33.421790 7664 raft_consensus.cc:348] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.421640 7663 raft_consensus.cc:348] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.422374 7663 raft_consensus.cc:374] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:33.422451 7664 raft_consensus.cc:374] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:33.422595 7664 raft_consensus.cc:729] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4453736ac41bf96f202ff66e9b31e, State: Initialized, Role: FOLLOWER
I20260928 10:50:33.422684 7663 raft_consensus.cc:729] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ec49710adf48f39982a06309948c60, State: Initialized, Role: FOLLOWER
I20260928 10:50:33.423435 7664 consensus_queue.cc:260] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.423774 7663 consensus_queue.cc:260] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.424961 7504 heartbeater.cc:503] Master 127.5.227.190:39843 was elected leader, sending a full tablet report...
I20260928 10:50:33.425354 7664 ts_tablet_manager.cc:1428] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 10:50:33.428736 7663 ts_tablet_manager.cc:1428] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Time spent starting tablet: real 0.019s user 0.013s sys 0.007s
I20260928 10:50:33.433267 7665 raft_consensus.cc:348] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.433849 7665 raft_consensus.cc:374] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:33.433986 7665 raft_consensus.cc:729] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ee37630bf1401a8f10034eb87e0646, State: Initialized, Role: FOLLOWER
I20260928 10:50:33.434722 7665 consensus_queue.cc:260] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.435876 7644 heartbeater.cc:503] Master 127.5.227.190:39843 was elected leader, sending a full tablet report...
I20260928 10:50:33.436467 7665 ts_tablet_manager.cc:1428] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Time spent starting tablet: real 0.020s user 0.010s sys 0.006s
W20260928 10:50:33.509727 7645 tablet.cc:2369] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:50:33.617137 7505 tablet.cc:2369] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:50:33.642884 7671 raft_consensus.cc:482] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:33.643539 7671 raft_consensus.cc:504] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.645668 7671 leader_election.cc:288] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130:43897), f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963)
I20260928 10:50:33.653363 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.646033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:56715 (local address 127.5.227.130:43897)
0928 10:50:33.646262 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.646268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.646295 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:50:33.646441 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.646449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.646627 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.646759 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:33.647258 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.648425 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.651214 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.652385 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.652599 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.652682 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.652944 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.653142 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:50:33.653388 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.645907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43897 (local address 127.5.227.131:56715)
0928 10:50:33.646115 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.646141 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.646287 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.646790 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.646797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.646836 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:33.647067 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.647082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.648624 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.648628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.651020 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.651037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.652616 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.652621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.652767 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.652794 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 10:50:33.652885 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 10:50:33.653045 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 10:50:33.654016 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.646429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:46701 (local address 127.5.227.129:37963)
0928 10:50:33.646663 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.646671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.647213 (+ 542us) server_negotiation.cc:407] Connection header received
0928 10:50:33.647286 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.647291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.647374 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.647543 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:33.648193 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.648999 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.651657 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.652909 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.653071 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.653138 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.653670 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.653808 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:50:33.653957 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.646265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37963 (local address 127.5.227.131:46701)
0928 10:50:33.647009 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.647046 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.647173 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.647509 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.647512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.647532 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:33.647911 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.647925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.649145 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.649150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.651478 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.651487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.653156 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.653162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.653334 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.653365 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:50:33.653470 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 10:50:33.653609 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":406,"threads_started":1}
I20260928 10:50:33.654706 7459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32283fce68dd4bcaa6b7d92d4cd4452b" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4453736ac41bf96f202ff66e9b31e" is_pre_election: true
I20260928 10:50:33.655193 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32283fce68dd4bcaa6b7d92d4cd4452b" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ec49710adf48f39982a06309948c60" is_pre_election: true
I20260928 10:50:33.655395 7459 raft_consensus.cc:2454] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 0.
I20260928 10:50:33.655764 7314 raft_consensus.cc:2454] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 0.
I20260928 10:50:33.656302 7524 leader_election.cc:302] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4453736ac41bf96f202ff66e9b31e, 34ee37630bf1401a8f10034eb87e0646; no voters:
I20260928 10:50:33.656998 7671 raft_consensus.cc:2781] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:33.657155 7671 raft_consensus.cc:482] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:33.657290 7671 raft_consensus.cc:3037] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:33.659639 7671 raft_consensus.cc:504] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.660620 7671 leader_election.cc:288] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 election: Requested vote from peers 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130:43897), f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963)
I20260928 10:50:33.661270 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32283fce68dd4bcaa6b7d92d4cd4452b" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ec49710adf48f39982a06309948c60"
I20260928 10:50:33.661288 7459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32283fce68dd4bcaa6b7d92d4cd4452b" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4453736ac41bf96f202ff66e9b31e"
I20260928 10:50:33.661526 7459 raft_consensus.cc:3037] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:33.661514 7314 raft_consensus.cc:3037] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:33.664034 7459 raft_consensus.cc:2454] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 1.
I20260928 10:50:33.664716 7314 raft_consensus.cc:2454] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 1.
I20260928 10:50:33.664708 7524 leader_election.cc:302] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4453736ac41bf96f202ff66e9b31e, 34ee37630bf1401a8f10034eb87e0646; no voters:
I20260928 10:50:33.665167 7671 raft_consensus.cc:2781] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:33.665654 7671 raft_consensus.cc:686] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 1 LEADER]: Becoming Leader. State: Replica: 34ee37630bf1401a8f10034eb87e0646, State: Running, Role: LEADER
I20260928 10:50:33.666158 7671 consensus_queue.cc:237] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.671327 7156 catalog_manager.cc:5495] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34ee37630bf1401a8f10034eb87e0646 (127.5.227.131). New cstate: current_term: 1 leader_uuid: "34ee37630bf1401a8f10034eb87e0646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } health_report { overall_health: HEALTHY } } }
W20260928 10:50:33.672806 7365 tablet.cc:2369] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:50:33.749874 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.743384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.5.227.131:41539)
0928 10:50:33.743685 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.743691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.743714 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:50:33.744151 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.744156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.744239 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.744422 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:33.744841 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.745610 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.747073 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.747530 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.747964 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.748544 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:33.749145 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:33.749325 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:33.749478 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 10:50:33.749573 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260928 10:50:33.760389 7459 raft_consensus.cc:1261] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Refusing update from remote peer 34ee37630bf1401a8f10034eb87e0646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:33.760391 7314 raft_consensus.cc:1261] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Refusing update from remote peer 34ee37630bf1401a8f10034eb87e0646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:33.761510 7673 consensus_queue.cc:1035] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:33.761941 7671 consensus_queue.cc:1035] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:33.775383 7676 mvcc.cc:205] Tried to move back new op lower bound from 7334267427870289920 to 7334267427500466176. Current Snapshot: MvccSnapshot[applied={T|T < 7334267427870289920}]
I20260928 10:50:33.799261 7157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:33.799893 7157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:33.804313 7157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973: Sending DeleteTablet for 3 replicas of tablet 32283fce68dd4bcaa6b7d92d4cd4452b
I20260928 10:50:33.805764 7434 tablet_service.cc:1514] Processing DeleteTablet for tablet 32283fce68dd4bcaa6b7d92d4cd4452b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:33 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:42532
I20260928 10:50:33.805891 7294 tablet_service.cc:1514] Processing DeleteTablet for tablet 32283fce68dd4bcaa6b7d92d4cd4452b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:33 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:52916
I20260928 10:50:33.806006 7574 tablet_service.cc:1514] Processing DeleteTablet for tablet 32283fce68dd4bcaa6b7d92d4cd4452b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:33 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:56220
I20260928 10:50:33.808164 7157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:50:33.809491 7157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:33.823153 7293 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe949842a2140edb56d4dce417bd081 (DEFAULT_TABLE table=test-table [id=bc3ece0606a242428032b6234365e45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:33.823608 7433 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe949842a2140edb56d4dce417bd081 (DEFAULT_TABLE table=test-table [id=bc3ece0606a242428032b6234365e45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:33.824079 7293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe949842a2140edb56d4dce417bd081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:33.824712 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe949842a2140edb56d4dce417bd081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:33.825969 7687 tablet_replica.cc:330] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: stopping tablet replica
I20260928 10:50:33.826959 7687 raft_consensus.cc:2229] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:33.827925 7687 raft_consensus.cc:2258] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:33.828054 7573 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe949842a2140edb56d4dce417bd081 (DEFAULT_TABLE table=test-table [id=bc3ece0606a242428032b6234365e45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:33.828912 7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe949842a2140edb56d4dce417bd081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:33.835301 7688 tablet_replica.cc:330] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: stopping tablet replica
I20260928 10:50:33.835888 7688 raft_consensus.cc:2229] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:50:33.836652 7687 ts_tablet_manager.cc:1905] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:33.837405 7688 raft_consensus.cc:2258] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:33.838883 7664 tablet_bootstrap.cc:492] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: Bootstrap starting.
I20260928 10:50:33.841249 7664 tablet_bootstrap.cc:654] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:33.843364 7688 ts_tablet_manager.cc:1905] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:33.843881 7665 tablet_bootstrap.cc:492] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: Bootstrap starting.
I20260928 10:50:33.844461 7663 tablet_bootstrap.cc:492] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: Bootstrap starting.
I20260928 10:50:33.846387 7663 tablet_bootstrap.cc:654] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:33.846662 7665 tablet_bootstrap.cc:654] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:33.859141 7687 ts_tablet_manager.cc:1918] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:50:33.868713 7686 tablet_replica.cc:330] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: stopping tablet replica
I20260928 10:50:33.869271 7686 raft_consensus.cc:2229] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:33.869866 7686 raft_consensus.cc:2258] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:33.869933 7665 tablet_bootstrap.cc:492] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: No bootstrap required, opened a new log
I20260928 10:50:33.870215 7665 ts_tablet_manager.cc:1397] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.002s
I20260928 10:50:33.871301 7687 log.cc:1196] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/32283fce68dd4bcaa6b7d92d4cd4452b
I20260928 10:50:33.871217 7665 raft_consensus.cc:348] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.871678 7665 raft_consensus.cc:374] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:33.871829 7664 tablet_bootstrap.cc:492] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: No bootstrap required, opened a new log
I20260928 10:50:33.871918 7687 ts_tablet_manager.cc:1939] T 32283fce68dd4bcaa6b7d92d4cd4452b P f8ec49710adf48f39982a06309948c60: Deleting consensus metadata
I20260928 10:50:33.871982 7664 ts_tablet_manager.cc:1397] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.008s
I20260928 10:50:33.871954 7665 raft_consensus.cc:729] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ee37630bf1401a8f10034eb87e0646, State: Initialized, Role: FOLLOWER
I20260928 10:50:33.872413 7665 consensus_queue.cc:260] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.872947 7664 raft_consensus.cc:348] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.873186 7664 raft_consensus.cc:374] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:33.873279 7664 raft_consensus.cc:729] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4453736ac41bf96f202ff66e9b31e, State: Initialized, Role: FOLLOWER
I20260928 10:50:33.873523 7664 consensus_queue.cc:260] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.873953 7686 ts_tablet_manager.cc:1905] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:33.870885 7663 tablet_bootstrap.cc:492] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: No bootstrap required, opened a new log
I20260928 10:50:33.874250 7663 ts_tablet_manager.cc:1397] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.004s
I20260928 10:50:33.874339 7144 catalog_manager.cc:4841] TS f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963): tablet 32283fce68dd4bcaa6b7d92d4cd4452b (table test-table [id=eabaa59151514b0e97bfa7d3e078ba71]) successfully deleted
I20260928 10:50:33.874574 7664 ts_tablet_manager.cc:1428] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:50:33.875078 7665 ts_tablet_manager.cc:1428] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 10:50:33.875118 7663 raft_consensus.cc:348] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.875344 7663 raft_consensus.cc:374] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:33.875440 7663 raft_consensus.cc:729] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ec49710adf48f39982a06309948c60, State: Initialized, Role: FOLLOWER
I20260928 10:50:33.875762 7663 consensus_queue.cc:260] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.876425 7663 ts_tablet_manager.cc:1428] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 10:50:33.876789 7688 ts_tablet_manager.cc:1918] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:50:33.877044 7688 log.cc:1196] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/32283fce68dd4bcaa6b7d92d4cd4452b
I20260928 10:50:33.877660 7688 ts_tablet_manager.cc:1939] T 32283fce68dd4bcaa6b7d92d4cd4452b P 34ee37630bf1401a8f10034eb87e0646: Deleting consensus metadata
I20260928 10:50:33.880511 7143 catalog_manager.cc:4841] TS 34ee37630bf1401a8f10034eb87e0646 (127.5.227.131:41539): tablet 32283fce68dd4bcaa6b7d92d4cd4452b (table test-table [id=eabaa59151514b0e97bfa7d3e078ba71]) successfully deleted
I20260928 10:50:33.883594 7686 ts_tablet_manager.cc:1918] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:50:33.883826 7686 log.cc:1196] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/32283fce68dd4bcaa6b7d92d4cd4452b
I20260928 10:50:33.884472 7686 ts_tablet_manager.cc:1939] T 32283fce68dd4bcaa6b7d92d4cd4452b P 1be4453736ac41bf96f202ff66e9b31e: Deleting consensus metadata
I20260928 10:50:33.886577 7141 catalog_manager.cc:4841] TS 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130:43897): tablet 32283fce68dd4bcaa6b7d92d4cd4452b (table test-table [id=eabaa59151514b0e97bfa7d3e078ba71]) successfully deleted
I20260928 10:50:33.935883 7673 raft_consensus.cc:482] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:33.936141 7673 raft_consensus.cc:504] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.936866 7673 leader_election.cc:288] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963), 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130:43897)
I20260928 10:50:33.937487 7459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe949842a2140edb56d4dce417bd081" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4453736ac41bf96f202ff66e9b31e" is_pre_election: true
I20260928 10:50:33.937784 7459 raft_consensus.cc:2454] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 0.
I20260928 10:50:33.937750 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe949842a2140edb56d4dce417bd081" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ec49710adf48f39982a06309948c60" is_pre_election: true
I20260928 10:50:33.937976 7314 raft_consensus.cc:2454] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 0.
I20260928 10:50:33.938406 7524 leader_election.cc:302] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4453736ac41bf96f202ff66e9b31e, 34ee37630bf1401a8f10034eb87e0646; no voters:
I20260928 10:50:33.938814 7673 raft_consensus.cc:2781] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:33.938933 7673 raft_consensus.cc:482] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:33.939051 7673 raft_consensus.cc:3037] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:33.942414 7673 raft_consensus.cc:504] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.943256 7673 leader_election.cc:288] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 election: Requested vote from peers f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963), 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130:43897)
I20260928 10:50:33.944334 7459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe949842a2140edb56d4dce417bd081" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be4453736ac41bf96f202ff66e9b31e"
I20260928 10:50:33.944344 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe949842a2140edb56d4dce417bd081" candidate_uuid: "34ee37630bf1401a8f10034eb87e0646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ec49710adf48f39982a06309948c60"
I20260928 10:50:33.944566 7459 raft_consensus.cc:3037] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:33.944584 7314 raft_consensus.cc:3037] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:33.946722 7459 raft_consensus.cc:2454] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 1.
I20260928 10:50:33.947395 7524 leader_election.cc:302] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4453736ac41bf96f202ff66e9b31e, 34ee37630bf1401a8f10034eb87e0646; no voters:
I20260928 10:50:33.947678 7314 raft_consensus.cc:2454] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ee37630bf1401a8f10034eb87e0646 in term 1.
I20260928 10:50:33.947846 7673 raft_consensus.cc:2781] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:33.948428 7673 raft_consensus.cc:686] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 1 LEADER]: Becoming Leader. State: Replica: 34ee37630bf1401a8f10034eb87e0646, State: Running, Role: LEADER
I20260928 10:50:33.948856 7673 consensus_queue.cc:237] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:33.952224 7155 catalog_manager.cc:5495] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34ee37630bf1401a8f10034eb87e0646 (127.5.227.131). New cstate: current_term: 1 leader_uuid: "34ee37630bf1401a8f10034eb87e0646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } health_report { overall_health: HEALTHY } } }
Sep 28 10:50:33 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592631, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:34.009492 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.990518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.5.227.190:39843)
0928 10:50:33.991508 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:33.991513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:33.991538 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:50:33.991651 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:33.991661 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:33.991715 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:33.991884 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:33.993297 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.994090 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.995688 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.996415 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.996745 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.001114 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:34.001120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:34.001123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:34.001158 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:34.005291 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.006142 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.006153 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.006157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.006295 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.007374 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.007381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.007389 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.007960 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:34.008148 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.008733 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.009031 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":848,"thread_start_us":705,"threads_started":1}
I20260928 10:50:34.009776 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:33.990466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39843 (local address 127.0.0.1:50934)
0928 10:50:33.991332 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:33.991357 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:33.991521 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:33.991907 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:33.991912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:33.992800 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:33.993044 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.993054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.994251 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.994258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.995354 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:33.995370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:33.996759 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:33.996765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:33.997088 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:33.997950 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:33.998025 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:34.000832 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:34.005473 (+ 4641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:34.005496 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:34.005510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:34.005977 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:34.006428 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:34.006433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:34.006435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:34.007216 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:34.008136 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:34.008147 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:34.008339 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 10:50:34.008647 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 10:50:34.009201 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":242,"thread_start_us":387,"threads_started":1}
I20260928 10:50:34.018268 7155 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:50934:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:50:34.019870 7155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:34.029990 7293 tablet_service.cc:1467] Processing CreateTablet for tablet d4d51b259cb240c3a82cacfe70d8f33f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=256ff3418b82496eb083078ef2caed37]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:50:34.030295 7574 tablet_service.cc:1467] Processing CreateTablet for tablet d4d51b259cb240c3a82cacfe70d8f33f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=256ff3418b82496eb083078ef2caed37]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:50:34.030628 7293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d51b259cb240c3a82cacfe70d8f33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:34.030865 7574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d51b259cb240c3a82cacfe70d8f33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:34.030782 7434 tablet_service.cc:1467] Processing CreateTablet for tablet d4d51b259cb240c3a82cacfe70d8f33f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=256ff3418b82496eb083078ef2caed37]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:50:34.031314 7434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d51b259cb240c3a82cacfe70d8f33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:34.039779 7663 tablet_bootstrap.cc:492] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: Bootstrap starting.
I20260928 10:50:34.039842 7665 tablet_bootstrap.cc:492] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: Bootstrap starting.
I20260928 10:50:34.041669 7663 tablet_bootstrap.cc:654] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:34.041698 7665 tablet_bootstrap.cc:654] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:34.042259 7664 tablet_bootstrap.cc:492] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: Bootstrap starting.
I20260928 10:50:34.044330 7664 tablet_bootstrap.cc:654] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:34.052302 7663 tablet_bootstrap.cc:492] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: No bootstrap required, opened a new log
I20260928 10:50:34.052544 7663 ts_tablet_manager.cc:1397] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260928 10:50:34.053190 7663 raft_consensus.cc:348] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.053418 7663 raft_consensus.cc:374] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:34.053509 7663 raft_consensus.cc:729] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ec49710adf48f39982a06309948c60, State: Initialized, Role: FOLLOWER
I20260928 10:50:34.054033 7663 consensus_queue.cc:260] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.054793 7670 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.055213 7670 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:34.056330 7663 ts_tablet_manager.cc:1428] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 10:50:34.057267 7664 tablet_bootstrap.cc:492] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: No bootstrap required, opened a new log
I20260928 10:50:34.057510 7665 tablet_bootstrap.cc:492] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: No bootstrap required, opened a new log
I20260928 10:50:34.057698 7665 ts_tablet_manager.cc:1397] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20260928 10:50:34.057893 7664 ts_tablet_manager.cc:1397] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260928 10:50:34.058527 7665 raft_consensus.cc:348] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.058825 7665 raft_consensus.cc:374] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:34.058913 7665 raft_consensus.cc:729] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ee37630bf1401a8f10034eb87e0646, State: Initialized, Role: FOLLOWER
I20260928 10:50:34.058881 7664 raft_consensus.cc:348] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.059278 7664 raft_consensus.cc:374] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:34.059216 7665 consensus_queue.cc:260] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.059398 7664 raft_consensus.cc:729] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be4453736ac41bf96f202ff66e9b31e, State: Initialized, Role: FOLLOWER
I20260928 10:50:34.059947 7665 ts_tablet_manager.cc:1428] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 10:50:34.060132 7673 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.060472 7673 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:34.060597 7664 consensus_queue.cc:260] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.061209 7664 ts_tablet_manager.cc:1428] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 10:50:34.061672 7669 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.062076 7669 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:34.081017 7669 raft_consensus.cc:482] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:34.081334 7669 raft_consensus.cc:504] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.082990 7669 leader_election.cc:288] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963), 34ee37630bf1401a8f10034eb87e0646 (127.5.227.131:41539)
I20260928 10:50:34.095294 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.083961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:40337 (local address 127.5.227.129:37963)
0928 10:50:34.084142 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.084147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.085008 (+ 861us) server_negotiation.cc:407] Connection header received
0928 10:50:34.085330 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.085338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.085418 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.085530 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:34.087304 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.088710 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.092158 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.093991 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.094242 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.094323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.094818 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.094957 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:50:34.095352 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.083390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37963 (local address 127.5.227.130:40337)
0928 10:50:34.084885 (+ 1495us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:34.084905 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:34.085116 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:34.085765 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:34.085773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:34.085800 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:34.087085 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.087096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.088932 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.088938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.091890 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.091913 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.094327 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.094334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.094599 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.094630 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:50:34.094785 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 10:50:34.094946 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":205,"thread_start_us":560,"threads_started":1}
I20260928 10:50:34.096472 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d51b259cb240c3a82cacfe70d8f33f" candidate_uuid: "1be4453736ac41bf96f202ff66e9b31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ec49710adf48f39982a06309948c60" is_pre_election: true
I20260928 10:50:34.096822 7314 raft_consensus.cc:2454] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4453736ac41bf96f202ff66e9b31e in term 0.
I20260928 10:50:34.097008 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.083240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:41539 (local address 127.5.227.130:53347)
0928 10:50:34.084323 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:34.084353 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:34.084605 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:34.085008 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:34.085012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:34.085038 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:34.086893 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.086910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.088932 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.088938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.091890 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.091913 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.096105 (+ 4192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.096113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.096321 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.096356 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:50:34.096467 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 10:50:34.096615 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":216,"thread_start_us":1118,"threads_started":1}
I20260928 10:50:34.097002 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.083536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:53347 (local address 127.5.227.131:41539)
0928 10:50:34.083801 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.083807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.084470 (+ 663us) server_negotiation.cc:407] Connection header received
0928 10:50:34.084768 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.084772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.084863 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.084978 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:34.087309 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.088732 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.094459 (+ 5727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.095771 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.095939 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.096012 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.096525 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.096700 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:50:34.097669 7387 leader_election.cc:302] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4453736ac41bf96f202ff66e9b31e, f8ec49710adf48f39982a06309948c60; no voters:
I20260928 10:50:34.098573 7669 raft_consensus.cc:2781] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:34.098721 7669 raft_consensus.cc:482] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:34.098805 7669 raft_consensus.cc:3037] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:34.098587 7594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d51b259cb240c3a82cacfe70d8f33f" candidate_uuid: "1be4453736ac41bf96f202ff66e9b31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ee37630bf1401a8f10034eb87e0646" is_pre_election: true
I20260928 10:50:34.099159 7594 raft_consensus.cc:2454] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be4453736ac41bf96f202ff66e9b31e in term 0.
I20260928 10:50:34.102103 7669 raft_consensus.cc:504] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.102905 7669 leader_election.cc:288] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [CANDIDATE]: Term 1 election: Requested vote from peers f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963), 34ee37630bf1401a8f10034eb87e0646 (127.5.227.131:41539)
I20260928 10:50:34.103694 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d51b259cb240c3a82cacfe70d8f33f" candidate_uuid: "1be4453736ac41bf96f202ff66e9b31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ec49710adf48f39982a06309948c60"
I20260928 10:50:34.103977 7314 raft_consensus.cc:3037] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:34.103924 7594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d51b259cb240c3a82cacfe70d8f33f" candidate_uuid: "1be4453736ac41bf96f202ff66e9b31e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ee37630bf1401a8f10034eb87e0646"
I20260928 10:50:34.104233 7594 raft_consensus.cc:3037] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:34.106688 7594 raft_consensus.cc:2454] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4453736ac41bf96f202ff66e9b31e in term 1.
I20260928 10:50:34.107272 7386 leader_election.cc:302] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be4453736ac41bf96f202ff66e9b31e, 34ee37630bf1401a8f10034eb87e0646; no voters:
I20260928 10:50:34.107442 7314 raft_consensus.cc:2454] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be4453736ac41bf96f202ff66e9b31e in term 1.
I20260928 10:50:34.107748 7669 raft_consensus.cc:2781] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:34.108368 7669 raft_consensus.cc:686] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [term 1 LEADER]: Becoming Leader. State: Replica: 1be4453736ac41bf96f202ff66e9b31e, State: Running, Role: LEADER
I20260928 10:50:34.108965 7669 consensus_queue.cc:237] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } }
I20260928 10:50:34.113013 7705 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: TxnStatusTablet state changed. Reason: New leader 1be4453736ac41bf96f202ff66e9b31e. Latest consensus state: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.113454 7705 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:50:34.114195 7707 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:50:34.113932 7157 catalog_manager.cc:5495] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130). New cstate: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } health_report { overall_health: UNKNOWN } } }
I20260928 10:50:34.114430 7707 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:50:34.159715 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.151722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.5.227.130:43897)
0928 10:50:34.151968 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.151976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.151997 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:50:34.152118 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.152124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.152250 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.152414 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:34.153284 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.154450 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.156660 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.157346 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.157496 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.158163 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:34.158951 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:34.159118 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.159273 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.159372 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:50:34.159799 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.150064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43897 (local address 127.0.0.1:42536)
0928 10:50:34.151072 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:34.151098 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:34.151290 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:34.152762 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:34.152768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:34.152786 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:50:34.153084 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.153105 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.154637 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.154644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.156448 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.156467 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.157543 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.157550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.157829 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.157917 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:50:34.159100 (+ 1183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:50:34.159126 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:50:34.159234 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 10:50:34.159419 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388}
I20260928 10:50:34.165529 7314 raft_consensus.cc:1261] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Refusing update from remote peer 1be4453736ac41bf96f202ff66e9b31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:34.165964 7594 raft_consensus.cc:1261] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Refusing update from remote peer 1be4453736ac41bf96f202ff66e9b31e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:34.166548 7669 consensus_queue.cc:1035] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:34.167270 7705 consensus_queue.cc:1035] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:34.182204 7670 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: TxnStatusTablet state changed. Reason: New leader 1be4453736ac41bf96f202ff66e9b31e. Latest consensus state: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.182574 7670 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:34.184345 7673 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: TxnStatusTablet state changed. Reason: New leader 1be4453736ac41bf96f202ff66e9b31e. Latest consensus state: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.184880 7673 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:34.186636 7705 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.186990 7705 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:50:34.188745 7677 mvcc.cc:205] Tried to move back new op lower bound from 7334267429532708864 to 7334267429315440640. Current Snapshot: MvccSnapshot[applied={T|T < 7334267429532708864}]
I20260928 10:50:34.189085 7670 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.189322 7670 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P f8ec49710adf48f39982a06309948c60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:34.192291 7669 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.192794 7669 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P 1be4453736ac41bf96f202ff66e9b31e: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:50:34.193002 7678 mvcc.cc:205] Tried to move back new op lower bound from 7334267429532708864 to 7334267429315440640. Current Snapshot: MvccSnapshot[applied={T|T < 7334267429532708864 or (T in {7334267429532708864})}]
I20260928 10:50:34.202275 7673 tablet_replica.cc:439] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be4453736ac41bf96f202ff66e9b31e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 } } peers { permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 } } peers { permanent_uuid: "34ee37630bf1401a8f10034eb87e0646" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41539 } } }
I20260928 10:50:34.202630 7673 tablet_replica.cc:442] T d4d51b259cb240c3a82cacfe70d8f33f P 34ee37630bf1401a8f10034eb87e0646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:34.222978 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.213821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:60577 (local address 127.5.227.130:43897)
0928 10:50:34.214051 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.214058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.214821 (+ 763us) server_negotiation.cc:407] Connection header received
0928 10:50:34.215153 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.215159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.215249 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.215327 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:34.216232 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.217401 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.220460 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.221191 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.221302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.222056 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:34.222212 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:34.222290 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.222529 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.222642 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:50:34.224315 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.213857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43897 (local address 127.5.227.131:60577)
0928 10:50:34.214694 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:34.214721 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:34.214975 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:34.215626 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:34.215631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:34.215652 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:50:34.216034 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.216046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.217717 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.217724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.220232 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.220257 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.221521 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.221528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.221726 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.221810 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:50:34.222377 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:50:34.222388 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:50:34.223505 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0928 10:50:34.223867 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":415,"threads_started":1}
I20260928 10:50:34.241281 7314 raft_consensus.cc:1261] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Refusing update from remote peer 34ee37630bf1401a8f10034eb87e0646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:34.241343 7459 raft_consensus.cc:1261] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Refusing update from remote peer 34ee37630bf1401a8f10034eb87e0646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:34.242164 7673 consensus_queue.cc:1035] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ec49710adf48f39982a06309948c60" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:34.242484 7671 consensus_queue.cc:1035] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be4453736ac41bf96f202ff66e9b31e" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:34.335031 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.301229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:42965 (local address 127.5.227.131:41539)
0928 10:50:34.301473 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.301479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.321792 (+ 20313us) server_negotiation.cc:407] Connection header received
0928 10:50:34.325669 (+ 3877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.325676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.325782 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.325852 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:34.326717 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.328112 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.331997 (+ 3885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.332879 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.333006 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.333854 (+ 848us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:34.334020 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:34.334141 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.334322 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.334450 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:50:34.335098 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.300671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:41539 (local address 127.5.227.130:42965)
0928 10:50:34.321651 (+ 20980us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:34.321677 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:34.321935 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:34.326107 (+ 4172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:34.326112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:34.326128 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:50:34.326514 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.326527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.328320 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.328328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.331220 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:34.331241 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.333230 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.333237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:34.333469 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.333559 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:50:34.334161 (+ 602us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:50:34.334174 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 10:50:34.334279 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 10:50:34.334472 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20740,"thread_start_us":20438,"threads_started":1}
I20260928 10:50:34.390090 7156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:34.390359 7156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:34.394546 7156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P add823eb42e948249afbc392ba2b7973: Sending DeleteTablet for 3 replicas of tablet 5fe949842a2140edb56d4dce417bd081
I20260928 10:50:34.395524 7293 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe949842a2140edb56d4dce417bd081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:34 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:52916
I20260928 10:50:34.395848 7434 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe949842a2140edb56d4dce417bd081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:34 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:42532
I20260928 10:50:34.396530 7574 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe949842a2140edb56d4dce417bd081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:34 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:56220
I20260928 10:50:34.418663 7723 tablet_replica.cc:330] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: stopping tablet replica
I20260928 10:50:34.419101 7723 raft_consensus.cc:2229] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:50:34.419699 7723 raft_consensus.cc:2258] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:34.420284 7721 tablet_replica.cc:330] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: stopping tablet replica
I20260928 10:50:34.421609 7723 ts_tablet_manager.cc:1905] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:34.422014 7721 raft_consensus.cc:2229] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:34.422528 7721 raft_consensus.cc:2258] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:34.426183 7721 ts_tablet_manager.cc:1905] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:34.426793 7722 tablet_replica.cc:330] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: stopping tablet replica
I20260928 10:50:34.427685 7722 raft_consensus.cc:2229] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:34.430907 7722 raft_consensus.cc:2258] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:34.432544 7722 ts_tablet_manager.cc:1905] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:34.438540 7723 ts_tablet_manager.cc:1918] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:34.439348 7723 log.cc:1196] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/5fe949842a2140edb56d4dce417bd081
I20260928 10:50:34.440431 7723 ts_tablet_manager.cc:1939] T 5fe949842a2140edb56d4dce417bd081 P 34ee37630bf1401a8f10034eb87e0646: Deleting consensus metadata
I20260928 10:50:34.441228 7721 ts_tablet_manager.cc:1918] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:34.443156 7143 catalog_manager.cc:4841] TS 34ee37630bf1401a8f10034eb87e0646 (127.5.227.131:41539): tablet 5fe949842a2140edb56d4dce417bd081 (table test-table [id=bc3ece0606a242428032b6234365e45a]) successfully deleted
I20260928 10:50:34.450166 7722 ts_tablet_manager.cc:1918] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:34.451021 7721 log.cc:1196] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/5fe949842a2140edb56d4dce417bd081
I20260928 10:50:34.451659 7721 ts_tablet_manager.cc:1939] T 5fe949842a2140edb56d4dce417bd081 P f8ec49710adf48f39982a06309948c60: Deleting consensus metadata
I20260928 10:50:34.453320 7722 log.cc:1196] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/5fe949842a2140edb56d4dce417bd081
I20260928 10:50:34.453604 7144 catalog_manager.cc:4841] TS f8ec49710adf48f39982a06309948c60 (127.5.227.129:37963): tablet 5fe949842a2140edb56d4dce417bd081 (table test-table [id=bc3ece0606a242428032b6234365e45a]) successfully deleted
I20260928 10:50:34.453981 7722 ts_tablet_manager.cc:1939] T 5fe949842a2140edb56d4dce417bd081 P 1be4453736ac41bf96f202ff66e9b31e: Deleting consensus metadata
I20260928 10:50:34.455442 7141 catalog_manager.cc:4841] TS 1be4453736ac41bf96f202ff66e9b31e (127.5.227.130:43897): tablet 5fe949842a2140edb56d4dce417bd081 (table test-table [id=bc3ece0606a242428032b6234365e45a]) successfully deleted
Sep 28 10:50:34 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:34.475776 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.427044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.5.227.129:37963)
0928 10:50:34.427249 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.427258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.451048 (+ 23790us) server_negotiation.cc:407] Connection header received
0928 10:50:34.451462 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.451468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.451808 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.451913 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:34.455470 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.456864 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.459649 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.460675 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.460851 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.466806 (+ 5955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:34.466827 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:34.466847 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:34.466905 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:34.471452 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.472752 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.472763 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.472768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.472961 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.473502 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.473510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.473515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.474196 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:34.474569 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.474939 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.475286 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 10:50:34.476711 7359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52920
I20260928 10:50:34.501616 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.481011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.5.227.190:39843)
0928 10:50:34.481244 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.481250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.481765 (+ 515us) server_negotiation.cc:407] Connection header received
0928 10:50:34.482093 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.482099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.482199 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.482280 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:34.485176 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.486518 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.489167 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.489874 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.490036 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.493857 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:34.493865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:34.493870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:34.493926 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:34.498033 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.499156 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.499163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.499167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.499355 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.499879 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.499885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.499891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.500321 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:34.500598 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.500949 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.501228 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 10:50:34.502369 7156 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50946
I20260928 10:50:34.504643 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7228
I20260928 10:50:34.524788 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.506514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.5.227.129:37963)
0928 10:50:34.506738 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.506746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.506770 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:50:34.506843 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.506849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.506932 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.507069 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:34.509355 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.510485 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.512482 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.513285 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.513442 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.516530 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:34.516539 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:34.516545 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:34.516601 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:34.521030 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.522068 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.522075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.522079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.522255 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.522645 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.522652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.522657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.523018 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:34.523301 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.523673 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.523968 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 10:50:34.525437 7359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52926
W20260928 10:50:34.526077 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7228: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:34.526291 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 7228
I20260928 10:50:34.545310 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7368
Sep 28 10:50:34 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:50:34.565689 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.546606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.5.227.130:43897)
0928 10:50:34.546792 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.546797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.546819 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:50:34.547045 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.547050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.547127 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.547222 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:34.549327 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.550133 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.552085 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.552924 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.553061 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.558717 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:34.558732 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:34.558748 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:34.558804 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:34.562409 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.563285 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.563293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.563296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.563431 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.564008 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.564022 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.564024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.564419 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:34.564779 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.565088 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.565339 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 10:50:34.566402 7499 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42544
W20260928 10:50:34.567349 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:34.567628 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 7368
I20260928 10:50:34.589794 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7508
Sep 28 10:50:34 dist-test-slave-8w8g krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:50:34.606760 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:34.590686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.5.227.131:41539)
0928 10:50:34.590893 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:34.590898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:34.590918 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:50:34.591056 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:34.591060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:34.591128 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:34.591215 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:34.592750 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.593514 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.595158 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:34.595603 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:34.595712 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:34.600500 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:34.600526 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:34.600543 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:34.600600 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:34.603893 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.604633 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.604641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.604644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.604789 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:34.605118 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:34.605121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:34.605123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:34.605545 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:34.605779 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:34.606166 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 10:50:34.606417 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 10:50:34.607419 7639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56232
W20260928 10:50:34.608232 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7508: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:34.608381 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 7508
I20260928 10:50:34.631000 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7125
I20260928 10:50:34.632409 7197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:34.750809 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 7125
2026-09-28T10:50:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5668 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:34 dist-test-slave-8w8g krb5kdc[7732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:34 dist-test-slave-8w8g krb5kdc[7732](info): set up 2 sockets
Sep 28 10:50:34 dist-test-slave-8w8g krb5kdc[7732](info): commencing operation
krb5kdc: starting...
W20260928 10:50:35.856405 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:35 dist-test-slave-8w8g krb5kdc[7732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:36.060966 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:46385
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:46385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:36.402246 7745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:36.402565 7745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:36.402725 7745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:36.410061 7745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:36.410175 7745 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:50:36.410254 7745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:36.410359 7745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:36.410416 7745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:36.410497 7745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:36.422194 7745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:46385
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:46385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:36.423635 7745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:36.425115 7745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:36.437537 7751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:36.437502 7750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:36.437552 7753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:36.438418 7745 server_base.cc:1034] running on GCE node
I20260928 10:50:36.439221 7745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:36.440794 7745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:36.442058 7745 hybrid_clock.cc:648] HybridClock initialized: now 1790592636441961 us; error 41 us; skew 500 ppm
I20260928 10:50:36.442425 7745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:36 dist-test-slave-8w8g krb5kdc[7732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592636, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:36.450035 7745 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:36.451618 7745 webserver.cc:460] Webserver started at http://127.5.227.190:35465/ using document root <none> and password file <none>
I20260928 10:50:36.452395 7745 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:36.452530 7745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:36.452862 7745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:36.455956 7745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2"
format_stamp: "Formatted at 2026-09-28 10:50:36 on dist-test-slave-8w8g"
server_key: "5dbbb29e0cfe899000f9ba32ce93413e"
server_key_iv: "a2702444b6c6876a1457ac9e61a735b5"
server_key_version: "encryptionkey@0"
I20260928 10:50:36.456748 7745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2"
format_stamp: "Formatted at 2026-09-28 10:50:36 on dist-test-slave-8w8g"
server_key: "5dbbb29e0cfe899000f9ba32ce93413e"
server_key_iv: "a2702444b6c6876a1457ac9e61a735b5"
server_key_version: "encryptionkey@0"
I20260928 10:50:36.462041 7745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 10:50:36.465776 7760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:36.467389 7745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:50:36.467635 7745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2"
format_stamp: "Formatted at 2026-09-28 10:50:36 on dist-test-slave-8w8g"
server_key: "5dbbb29e0cfe899000f9ba32ce93413e"
server_key_iv: "a2702444b6c6876a1457ac9e61a735b5"
server_key_version: "encryptionkey@0"
I20260928 10:50:36.467954 7745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:36.488068 7745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:36.492966 7745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:36.493271 7745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:36.517963 7745 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:46385
I20260928 10:50:36.517970 7821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:46385 every 8 connection(s)
I20260928 10:50:36.519464 7745 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:36.521574 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 7745
I20260928 10:50:36.521802 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:36.522444 6030 external_mini_cluster.cc:1428] Setting key 779198b426d4a3ba2ad39018e4b96b14
I20260928 10:50:36.524056 7822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:36.539503 7822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Bootstrap starting.
Sep 28 10:50:36 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:36.543722 7822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:36.545010 7822 log.cc:824] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:36.548755 7822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: No bootstrap required, opened a new log
I20260928 10:50:36.549963 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:36.524746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.5.227.190:46385)
0928 10:50:36.525580 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:36.525603 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:36.525646 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:50:36.527283 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:36.527307 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:36.528154 (+ 847us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:36.528547 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:36.530557 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:36.532143 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:36.534063 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:36.535161 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:36.535346 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:36.541321 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:36.541331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:36.541354 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:36.541425 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:36.545131 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:36.546192 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:36.546208 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:36.546217 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:36.546511 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:36.547006 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:36.547012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:36.547015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:36.547644 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:36.547971 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:36.548426 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 10:50:36.548862 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":386,"threads_started":1}
I20260928 10:50:36.556483 7822 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46385 } }
I20260928 10:50:36.556924 7822 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:36.557080 7822 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fcbbfa2c047c8b65cfc447cd66cb2, State: Initialized, Role: FOLLOWER
I20260928 10:50:36.557739 7822 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46385 } }
I20260928 10:50:36.558000 7822 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:36.558174 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:36.558400 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:36.590664 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46385 } }
I20260928 10:50:36.591463 7822 leader_election.cc:302] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe8fcbbfa2c047c8b65cfc447cd66cb2; no voters:
I20260928 10:50:36.592448 7822 leader_election.cc:288] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:36.592602 7827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:36.594223 7827 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [term 1 LEADER]: Becoming Leader. State: Replica: fe8fcbbfa2c047c8b65cfc447cd66cb2, State: Running, Role: LEADER
I20260928 10:50:36.594906 7827 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46385 } }
I20260928 10:50:36.596033 7822 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:36.598131 7829 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe8fcbbfa2c047c8b65cfc447cd66cb2. Latest consensus state: current_term: 1 leader_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46385 } } }
I20260928 10:50:36.598160 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8fcbbfa2c047c8b65cfc447cd66cb2" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 46385 } } }
I20260928 10:50:36.598658 7829 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:36.598658 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:36.600831 7833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:36.607372 7833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:36.617162 7833 catalog_manager.cc:1347] Generated new cluster ID: f9237c32c3004707ad8d82ce8d54c59f
I20260928 10:50:36.617300 7833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:36.635166 7833 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:36.636691 7833 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:36.644697 7833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Generated new TSK 0
I20260928 10:50:36.645750 7833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
W20260928 10:50:37.630314 6030 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.227.129: real 0.980s user 0.000s sys 0.012s
I20260928 10:50:37.631456 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46385
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:38.006027 7848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:38.006459 7848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:38.006579 7848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:38.015048 7848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:38.015241 7848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:38.015334 7848 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:50:38.015391 7848 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:50:38.015455 7848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:50:38.028031 7848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:46385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:38.029768 7848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:38.031883 7848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:38.048293 7853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:38.048620 7854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:38.048215 7856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:38.049994 7848 server_base.cc:1034] running on GCE node
I20260928 10:50:38.050745 7848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:38.051966 7848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:38.053511 7848 hybrid_clock.cc:648] HybridClock initialized: now 1790592638053452 us; error 71 us; skew 500 ppm
I20260928 10:50:38.053922 7848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:38 dist-test-slave-8w8g krb5kdc[7732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592638, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:38.057988 7848 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:50:38.060139 7848 webserver.cc:460] Webserver started at http://127.5.227.129:38537/ using document root <none> and password file <none>
I20260928 10:50:38.060948 7848 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:38.061141 7848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:38.061503 7848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:38.065130 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "290f0ca7161744a19fb563f8db1551ae"
format_stamp: "Formatted at 2026-09-28 10:50:38 on dist-test-slave-8w8g"
server_key: "c6b7f960e7e3d2caad4849453186f6de"
server_key_iv: "160efa5fbca3c9bb210c8f2078afcabf"
server_key_version: "encryptionkey@0"
I20260928 10:50:38.066021 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "290f0ca7161744a19fb563f8db1551ae"
format_stamp: "Formatted at 2026-09-28 10:50:38 on dist-test-slave-8w8g"
server_key: "c6b7f960e7e3d2caad4849453186f6de"
server_key_iv: "160efa5fbca3c9bb210c8f2078afcabf"
server_key_version: "encryptionkey@0"
I20260928 10:50:38.072211 7848 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:50:38.076607 7863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:38.078554 7848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:50:38.078919 7848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "290f0ca7161744a19fb563f8db1551ae"
format_stamp: "Formatted at 2026-09-28 10:50:38 on dist-test-slave-8w8g"
server_key: "c6b7f960e7e3d2caad4849453186f6de"
server_key_iv: "160efa5fbca3c9bb210c8f2078afcabf"
server_key_version: "encryptionkey@0"
I20260928 10:50:38.079192 7848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:38.095707 7848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:38.102085 7848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:38.102407 7848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:38.105903 7848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:38.106076 7848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:38.106215 7848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:38.106315 7848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:50:38 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592638, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:38.135164 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:38.106913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:57083 (local address 127.5.227.190:46385)
0928 10:50:38.107538 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:38.107549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:38.109125 (+ 1576us) server_negotiation.cc:407] Connection header received
0928 10:50:38.111552 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:38.111558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:38.111654 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:38.111848 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:38.114465 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.115965 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.117855 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.118666 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.118843 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:38.125480 (+ 6637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:38.125490 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:38.125496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:38.125563 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:38.130235 (+ 4672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:38.131605 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:38.131615 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:38.131621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:38.131865 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:38.132598 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:38.132608 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:38.132614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:38.133109 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:38.133306 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:38.134232 (+ 926us) server_negotiation.cc:299] Negotiation successful
0928 10:50:38.134574 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
I20260928 10:50:38.135998 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:38.107910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46385 (local address 127.5.227.129:57083)
0928 10:50:38.108857 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:38.108937 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:38.111144 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:38.112345 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:38.112361 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:38.113351 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:38.114162 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:38.114185 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.116203 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.116208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:38.117654 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:38.117670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.118934 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.118940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:38.119167 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:38.120316 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:38.120357 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:38.125083 (+ 4726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:38.130504 (+ 5421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:38.130515 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:38.130532 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:38.131448 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:38.132036 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:38.132044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:38.132051 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:38.132336 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:38.133324 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:38.133335 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:38.133791 (+ 456us) client_negotiation.cc:769] Sending connection context
0928 10:50:38.134056 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 10:50:38.134506 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":406,"threads_started":1}
I20260928 10:50:38.163580 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:38.143531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:45399 (local address 127.5.227.190:46385)
0928 10:50:38.143869 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:38.143878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:38.145129 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 10:50:38.145392 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:38.145400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:38.145499 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:38.145622 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:38.147497 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.148665 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.151092 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.151802 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.151956 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:38.155285 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:38.155291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:38.155294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:38.155331 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:38.159566 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:38.160887 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:38.160896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:38.160903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:38.161079 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:38.161658 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:38.161662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:38.161664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:38.162026 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:38.162178 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:38.163109 (+ 931us) server_negotiation.cc:299] Negotiation successful
0928 10:50:38.163311 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260928 10:50:38.164295 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:38.143604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46385 (local address 127.5.227.129:45399)
0928 10:50:38.144888 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:38.144984 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:38.145215 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:38.145808 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:38.145812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:38.146720 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:38.147285 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:38.147302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.148865 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.148870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:38.150815 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:38.150832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:38.152249 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:38.152257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:38.152501 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:38.153397 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:38.153431 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:38.155006 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:38.159824 (+ 4818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:38.159831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:38.159836 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:38.160690 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:38.161255 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:38.161263 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:38.161270 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:38.161505 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:38.162186 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:38.162194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:38.162693 (+ 499us) client_negotiation.cc:769] Sending connection context
0928 10:50:38.162938 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 10:50:38.163484 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"thread_start_us":916,"threads_started":1}
I20260928 10:50:38.170786 7848 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:43401
I20260928 10:50:38.170866 7983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:43401 every 8 connection(s)
I20260928 10:50:38.172313 7848 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:50:38.174587 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 7848
I20260928 10:50:38.174839 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:50:38.175381 7984 heartbeater.cc:347] Connected to a master server at 127.5.227.190:46385
I20260928 10:50:38.175673 7984 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:38.175678 6030 external_mini_cluster.cc:1428] Setting key ec9dd34acdc9f8e08762636f1bacdcf4
I20260928 10:50:38.177027 7984 heartbeater.cc:511] Master 127.5.227.190:46385 requested a full tablet report, sending...
I20260928 10:50:38.179505 7777 ts_manager.cc:194] Registered new tserver with Master: 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401)
I20260928 10:50:38.181639 7777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:57083
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:50:39.186309 7984 heartbeater.cc:503] Master 127.5.227.190:46385 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
W20260928 10:50:39.842402 6030 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.227.130: real 1.662s user 0.000s sys 0.010s
I20260928 10:50:39.843137 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46385
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:40.253954 7988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:40.254271 7988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:40.254374 7988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:40.263036 7988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:40.263203 7988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:40.263387 7988 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:50:40.263507 7988 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:50:40.263582 7988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:50:40.275590 7988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:46385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:40.277372 7988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:40.279392 7988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 10:50:40.298157 7988 server_base.cc:1034] running on GCE node
W20260928 10:50:40.298063 7994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:40.298063 7996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:40.298760 7993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:40.299899 7988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:40.301204 7988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:40.304679 7988 hybrid_clock.cc:648] HybridClock initialized: now 1790592640304550 us; error 98 us; skew 500 ppm
I20260928 10:50:40.305285 7988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:40 dist-test-slave-8w8g krb5kdc[7732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592640, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:40.310082 7988 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:50:40.312099 7988 webserver.cc:460] Webserver started at http://127.5.227.130:44127/ using document root <none> and password file <none>
I20260928 10:50:40.312948 7988 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:40.313122 7988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:40.313498 7988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:40.316917 7988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "2813f578a3fd4e48937a99d600405061"
format_stamp: "Formatted at 2026-09-28 10:50:40 on dist-test-slave-8w8g"
server_key: "b75fd7ec3f54bceda0427da3291caacb"
server_key_iv: "bb7701d7dfe74fc9b34318bf03d0ebea"
server_key_version: "encryptionkey@0"
I20260928 10:50:40.317792 7988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "2813f578a3fd4e48937a99d600405061"
format_stamp: "Formatted at 2026-09-28 10:50:40 on dist-test-slave-8w8g"
server_key: "b75fd7ec3f54bceda0427da3291caacb"
server_key_iv: "bb7701d7dfe74fc9b34318bf03d0ebea"
server_key_version: "encryptionkey@0"
I20260928 10:50:40.324400 7988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:50:40.329397 8003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:40.331457 7988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:50:40.331688 7988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "2813f578a3fd4e48937a99d600405061"
format_stamp: "Formatted at 2026-09-28 10:50:40 on dist-test-slave-8w8g"
server_key: "b75fd7ec3f54bceda0427da3291caacb"
server_key_iv: "bb7701d7dfe74fc9b34318bf03d0ebea"
server_key_version: "encryptionkey@0"
I20260928 10:50:40.331964 7988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:40.350288 7988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:40.356320 7988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:40.356693 7988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:40.361071 7988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:40.361238 7988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:40.361392 7988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:40.361526 7988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 10:50:40 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592640, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:40.396716 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:40.361165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:53939 (local address 127.5.227.190:46385)
0928 10:50:40.361722 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:40.361730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:40.363260 (+ 1530us) server_negotiation.cc:407] Connection header received
0928 10:50:40.367367 (+ 4107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:40.367374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:40.367506 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:40.367637 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:40.371284 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.372645 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.376656 (+ 4011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.377680 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.377971 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:40.385580 (+ 7609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:40.385589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:40.385596 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:40.385654 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:40.390343 (+ 4689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:40.391515 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:40.391522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:40.391526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:40.391773 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:40.392578 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:40.392587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:40.392599 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:40.393127 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:40.393526 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:40.395690 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0928 10:50:40.396057 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":296,"threads_started":1}
I20260928 10:50:40.398392 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:40.361962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46385 (local address 127.5.227.130:53939)
0928 10:50:40.363024 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:40.363083 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:40.366731 (+ 3648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:40.368251 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:40.368270 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:40.369488 (+ 1218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:40.370953 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:40.370997 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.373148 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.373152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:40.376255 (+ 3103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:40.376290 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.378082 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.378091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:40.378381 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:40.379849 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:40.379880 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:40.385141 (+ 5261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:40.390600 (+ 5459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:40.390611 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:40.390628 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:40.391266 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:40.391936 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:40.391941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:40.391945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:40.392346 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:40.393602 (+ 1256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:40.393616 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:40.394867 (+ 1251us) client_negotiation.cc:769] Sending connection context
0928 10:50:40.395415 (+ 548us) client_negotiation.cc:240] Negotiation successful
0928 10:50:40.396041 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":506,"threads_started":1}
I20260928 10:50:40.425009 7988 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:46479
I20260928 10:50:40.425011 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:46479 every 8 connection(s)
I20260928 10:50:40.427014 7988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:50:40.429620 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 7988
I20260928 10:50:40.429831 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:50:40.429808 8124 heartbeater.cc:347] Connected to a master server at 127.5.227.190:46385
I20260928 10:50:40.430162 8124 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:40.430485 6030 external_mini_cluster.cc:1428] Setting key 9d75fdc6157e96c78a685789033680e1
I20260928 10:50:40.431368 8124 heartbeater.cc:511] Master 127.5.227.190:46385 requested a full tablet report, sending...
I20260928 10:50:40.439142 7777 ts_manager.cc:194] Registered new tserver with Master: 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479)
I20260928 10:50:40.440215 7777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:53939
I20260928 10:50:40.441612 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:40.408563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46385 (local address 127.5.227.130:58717)
0928 10:50:40.420538 (+ 11975us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:40.420574 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:40.420841 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:40.421431 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:40.421436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:40.422086 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:40.422627 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:40.422652 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.424818 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.424826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:40.426699 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:40.426718 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.428051 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.428062 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:40.428351 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:40.429262 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:40.429295 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:40.430583 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:40.437469 (+ 6886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:40.437475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:40.437478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:40.438278 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:40.439033 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:40.439037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:40.439040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:40.439177 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:40.440112 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:40.440118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:40.440352 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 10:50:40.440522 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 10:50:40.440991 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11711,"thread_start_us":11504,"threads_started":1}
I20260928 10:50:40.442682 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:40.408652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:58717 (local address 127.5.227.190:46385)
0928 10:50:40.408954 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:40.408965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:40.420820 (+ 11855us) server_negotiation.cc:407] Connection header received
0928 10:50:40.421093 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:40.421101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:40.421230 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:40.421549 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:40.422923 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.424227 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.427019 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:40.427724 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:40.427962 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:40.430901 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:40.430910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:40.430918 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:40.430969 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:40.437237 (+ 6268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:40.438648 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:40.438656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:40.438660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:40.438864 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:40.439418 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:40.439426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:40.439430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:40.439946 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:40.440126 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:40.441943 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0928 10:50:40.442252 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:50:41.445560 8124 heartbeater.cc:503] Master 127.5.227.190:46385 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
W20260928 10:50:41.480243 6030 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.227.131: real 1.044s user 0.000s sys 0.012s
I20260928 10:50:41.481153 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:46385
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:41.866714 8128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:41.867327 8128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:41.867626 8128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:41.875922 8128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:41.876124 8128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:41.876292 8128 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:50:41.876384 8128 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:50:41.876696 8128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:50:41.888063 8128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:46385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:41.889492 8128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:41.890877 8128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:41.907142 8133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:41.907243 8134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:41.907271 8136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:41.908064 8128 server_base.cc:1034] running on GCE node
I20260928 10:50:41.908761 8128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:41.910122 8128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:41.911409 8128 hybrid_clock.cc:648] HybridClock initialized: now 1790592641911417 us; error 92 us; skew 500 ppm
I20260928 10:50:41.911927 8128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:41 dist-test-slave-8w8g krb5kdc[7732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592641, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:41.920864 8128 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:50:41.923399 8128 webserver.cc:460] Webserver started at http://127.5.227.131:36193/ using document root <none> and password file <none>
I20260928 10:50:41.924382 8128 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:41.924577 8128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:41.924971 8128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:41.929339 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "a8d9ab7146de4910b532b52d842b5777"
format_stamp: "Formatted at 2026-09-28 10:50:41 on dist-test-slave-8w8g"
server_key: "2725f6b7c02311f04e39f168265b0bf3"
server_key_iv: "525bc38304a9efcfc3660c2e873f9c6a"
server_key_version: "encryptionkey@0"
I20260928 10:50:41.930403 8128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "a8d9ab7146de4910b532b52d842b5777"
format_stamp: "Formatted at 2026-09-28 10:50:41 on dist-test-slave-8w8g"
server_key: "2725f6b7c02311f04e39f168265b0bf3"
server_key_iv: "525bc38304a9efcfc3660c2e873f9c6a"
server_key_version: "encryptionkey@0"
I20260928 10:50:41.940173 8128 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260928 10:50:41.947211 8143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:41.949775 8128 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:50:41.950133 8128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "a8d9ab7146de4910b532b52d842b5777"
format_stamp: "Formatted at 2026-09-28 10:50:41 on dist-test-slave-8w8g"
server_key: "2725f6b7c02311f04e39f168265b0bf3"
server_key_iv: "525bc38304a9efcfc3660c2e873f9c6a"
server_key_version: "encryptionkey@0"
I20260928 10:50:41.950398 8128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:41.974886 8128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:41.980469 8128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:41.980748 8128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:41.984441 8128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:41.984560 8128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:41.984701 8128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:41.984789 8128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:50:42 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592641, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:42.018630 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:41.985317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:37857 (local address 127.5.227.190:46385)
0928 10:50:41.985970 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:41.985977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:41.987410 (+ 1433us) server_negotiation.cc:407] Connection header received
0928 10:50:41.989990 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:41.989995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:41.990137 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:41.990256 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:41.993754 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:41.995376 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:41.999137 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.000108 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.000337 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.008373 (+ 8036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:42.008382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:42.008387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:42.008441 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:42.012504 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.014173 (+ 1669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.014178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.014180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.014533 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.015410 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.015415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.015417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.015925 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:42.016094 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.017849 (+ 1755us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.018120 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":367,"threads_started":1}
I20260928 10:50:42.021304 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:41.986071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46385 (local address 127.5.227.131:37857)
0928 10:50:41.987147 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:41.987200 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:41.989467 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:41.990938 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:41.990952 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:41.992044 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:41.993395 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:41.993423 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:41.995621 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:41.995627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:41.998833 (+ 3206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:41.998862 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.000716 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.000730 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.001007 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.002114 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:42.002157 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:42.007712 (+ 5555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:42.012757 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:42.012768 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:42.012794 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:42.013943 (+ 1149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:42.014728 (+ 785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:42.014734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:42.014738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:42.015203 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:42.016094 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:42.016110 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:42.016933 (+ 823us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.018945 (+ 2012us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.019719 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":437,"threads_started":1}
I20260928 10:50:42.038774 8128 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:34547
I20260928 10:50:42.038794 8263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:34547 every 8 connection(s)
I20260928 10:50:42.040668 8128 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:50:42.044041 8264 heartbeater.cc:347] Connected to a master server at 127.5.227.190:46385
I20260928 10:50:42.044555 8264 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:42.045120 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 8128
I20260928 10:50:42.045375 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:50:42.045516 8264 heartbeater.cc:511] Master 127.5.227.190:46385 requested a full tablet report, sending...
I20260928 10:50:42.045986 6030 external_mini_cluster.cc:1428] Setting key 0d0fdc9dea093bda6413db420c7121d9
I20260928 10:50:42.047191 7777 ts_manager.cc:194] Registered new tserver with Master: a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:42.048393 7777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:37857
I20260928 10:50:42.049857 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:50:42.050628 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.030627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46385 (local address 127.5.227.131:41589)
0928 10:50:42.031314 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.031337 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.031939 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.033025 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.033032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.034748 (+ 1716us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:42.035343 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.035359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.036682 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.036688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.038621 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.038644 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.039617 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.039625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.039862 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.040612 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:42.040632 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:42.042201 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:42.047238 (+ 5037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:42.047242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:42.047280 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:42.047842 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:42.048399 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:42.048403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:42.048407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:42.048567 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:42.049370 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:42.049377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:42.049604 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.049787 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.050164 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":367,"threads_started":1}
I20260928 10:50:42.050756 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.030425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:41589 (local address 127.5.227.190:46385)
0928 10:50:42.030695 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.030705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.031827 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 10:50:42.032205 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.032216 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.032697 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.032892 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:42.035579 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.036511 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.038827 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.039355 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.039521 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.042878 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:42.042887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:42.042891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:42.042943 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:42.047037 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.048001 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.048008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.048011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.048207 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.048705 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.048710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.048713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.049136 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:42.049279 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.050104 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.050439 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
Sep 28 10:50:42 dist-test-slave-8w8g krb5kdc[7732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:50:42 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:42.113830 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.092792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.5.227.190:46385)
0928 10:50:42.093139 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.093145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.093466 (+ 321us) server_negotiation.cc:407] Connection header received
0928 10:50:42.093693 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.093699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.093782 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.093911 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:42.096018 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.097402 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.099444 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.100007 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.100132 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.105640 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:42.105646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:42.105650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:42.105698 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:42.109781 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.110994 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.110999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.111002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.111162 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.111601 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.111605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.111607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.112072 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:42.112260 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.112949 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.113527 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260928 10:50:42.120888 7776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:50:42.129196 7776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:42.176545 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.148858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.5.227.129:43401)
0928 10:50:42.165812 (+ 16954us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.165822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.165855 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:50:42.165996 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.166006 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.166319 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.166606 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:42.167394 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.169014 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.172088 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.174670 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.174850 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.175126 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.175967 (+ 841us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.176165 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16772,"thread_start_us":16488,"threads_started":1}
I20260928 10:50:42.176878 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.147979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:43401 (local address 127.0.0.1:32954)
0928 10:50:42.164872 (+ 16893us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.164900 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.165075 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.166621 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.166629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.166662 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:42.167147 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.167162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.169168 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.169173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.171853 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.171878 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.174970 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.174976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.175237 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.175506 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.175912 (+ 406us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.176139 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16674,"thread_start_us":17214,"threads_started":1}
I20260928 10:50:42.178483 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.147883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.5.227.130:46479)
0928 10:50:42.167824 (+ 19941us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.167831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.167857 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:50:42.167938 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.167943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.168161 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.168412 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:42.169099 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.170585 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.174666 (+ 4081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.176434 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.176600 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.176853 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.177857 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.178059 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19752,"thread_start_us":19609,"threads_started":1}
I20260928 10:50:42.179414 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.147432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:46479 (local address 127.0.0.1:43190)
0928 10:50:42.163286 (+ 15854us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.163347 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.163749 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.168439 (+ 4690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.168444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.168471 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:42.168883 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.168897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.170796 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.170802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.174188 (+ 3386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.174210 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.177121 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.177131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.177359 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.177454 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.178891 (+ 1437us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.179044 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15461,"thread_start_us":16664,"threads_started":1}
I20260928 10:50:42.183655 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.148350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:34547 (local address 127.0.0.1:59928)
0928 10:50:42.165425 (+ 17075us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.165443 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.165584 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.168123 (+ 2539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.168129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.168155 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:42.174251 (+ 6096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.174265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.177253 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.177259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.180127 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.180143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.182421 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.182429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.182692 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.182729 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.182838 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.183004 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16912,"mutex_wait_us":5628,"thread_start_us":16760,"threads_started":1}
I20260928 10:50:42.183802 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.149192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.5.227.131:34547)
0928 10:50:42.167396 (+ 18204us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.167411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.167446 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:50:42.167558 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.167565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.167874 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.168123 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:42.174671 (+ 6548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.177047 (+ 2376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.180331 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.182101 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.182284 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.182754 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.182924 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.183157 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18007,"thread_start_us":17832,"threads_started":1}
I20260928 10:50:42.182786 7914 tablet_service.cc:1467] Processing CreateTablet for tablet ea415151cdb24c1f8745406f434dfbcf (DEFAULT_TABLE table=test-table [id=aee1f2bf9d6240efa91a81e29cad798a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:42.185099 8054 tablet_service.cc:1467] Processing CreateTablet for tablet ea415151cdb24c1f8745406f434dfbcf (DEFAULT_TABLE table=test-table [id=aee1f2bf9d6240efa91a81e29cad798a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:42.185708 7914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea415151cdb24c1f8745406f434dfbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.186887 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea415151cdb24c1f8745406f434dfbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.189363 8194 tablet_service.cc:1467] Processing CreateTablet for tablet ea415151cdb24c1f8745406f434dfbcf (DEFAULT_TABLE table=test-table [id=aee1f2bf9d6240efa91a81e29cad798a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:42.191138 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea415151cdb24c1f8745406f434dfbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.200958 8283 tablet_bootstrap.cc:492] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Bootstrap starting.
I20260928 10:50:42.204828 8284 tablet_bootstrap.cc:492] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Bootstrap starting.
I20260928 10:50:42.204876 8283 tablet_bootstrap.cc:654] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.206627 8283 log.cc:824] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:42.208815 8284 tablet_bootstrap.cc:654] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.209794 8286 tablet_bootstrap.cc:492] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Bootstrap starting.
I20260928 10:50:42.210245 8284 log.cc:824] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:42.211630 8283 tablet_bootstrap.cc:492] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: No bootstrap required, opened a new log
I20260928 10:50:42.212023 8283 ts_tablet_manager.cc:1397] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260928 10:50:42.213666 8286 tablet_bootstrap.cc:654] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.213791 8284 tablet_bootstrap.cc:492] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: No bootstrap required, opened a new log
I20260928 10:50:42.214092 8284 ts_tablet_manager.cc:1397] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 10:50:42.215332 8286 log.cc:824] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:42.218740 8286 tablet_bootstrap.cc:492] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: No bootstrap required, opened a new log
I20260928 10:50:42.219091 8286 ts_tablet_manager.cc:1397] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 10:50:42.225534 8284 raft_consensus.cc:348] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.225534 8283 raft_consensus.cc:348] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.226090 8283 raft_consensus.cc:374] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.226091 8284 raft_consensus.cc:374] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.226217 8284 raft_consensus.cc:729] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.226212 8283 raft_consensus.cc:729] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f0ca7161744a19fb563f8db1551ae, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.227185 8283 consensus_queue.cc:260] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.227185 8284 consensus_queue.cc:260] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.229791 8283 ts_tablet_manager.cc:1428] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Time spent starting tablet: real 0.018s user 0.014s sys 0.001s
I20260928 10:50:42.228796 8286 raft_consensus.cc:348] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.231285 8284 ts_tablet_manager.cc:1428] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260928 10:50:42.230448 8286 raft_consensus.cc:374] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.231710 8286 raft_consensus.cc:729] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ab7146de4910b532b52d842b5777, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.232496 8286 consensus_queue.cc:260] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.235383 8264 heartbeater.cc:503] Master 127.5.227.190:46385 was elected leader, sending a full tablet report...
I20260928 10:50:42.235375 8286 ts_tablet_manager.cc:1428] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260928 10:50:42.295495 8265 tablet.cc:2369] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:50:42.310853 8290 raft_consensus.cc:482] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:42.311322 8290 raft_consensus.cc:504] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.314725 8290 leader_election.cc:288] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:42.322355 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.315621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:38441 (local address 127.5.227.131:34547)
0928 10:50:42.315855 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.315860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.316498 (+ 638us) server_negotiation.cc:407] Connection header received
0928 10:50:42.316719 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.316723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.316784 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.316891 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:42.317623 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.318337 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.320491 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.321308 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.321533 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.321614 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.321904 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.322041 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:50:42.322757 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.315334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:34547 (local address 127.5.227.129:38441)
0928 10:50:42.316372 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.316391 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.316561 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.316923 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.316928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.316952 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:42.317441 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.317451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.318512 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.318518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.320309 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.320323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.321549 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.321556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.321730 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.321756 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.321876 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.322025 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":64,"thread_start_us":601,"threads_started":1}
I20260928 10:50:42.322914 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.314980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:54041 (local address 127.5.227.130:46479)
0928 10:50:42.315204 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.315210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.316315 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 10:50:42.316697 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.316707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.316774 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.316942 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:42.317544 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.318282 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.320516 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.321359 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.321811 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.321905 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.322263 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.322648 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:50:42.323087 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.315029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:46479 (local address 127.5.227.129:54041)
0928 10:50:42.316152 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.316176 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.316425 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.316909 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.316913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.316938 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:42.317364 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.317377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.318513 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.318517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.320323 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.320337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.321838 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.321849 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.322043 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.322068 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.322219 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.322673 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":69,"thread_start_us":632,"threads_started":1}
I20260928 10:50:42.324160 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea415151cdb24c1f8745406f434dfbcf" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777" is_pre_election: true
I20260928 10:50:42.324529 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea415151cdb24c1f8745406f434dfbcf" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f578a3fd4e48937a99d600405061" is_pre_election: true
I20260928 10:50:42.324851 8214 raft_consensus.cc:2454] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 0.
I20260928 10:50:42.325101 8074 raft_consensus.cc:2454] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 0.
I20260928 10:50:42.326001 7865 leader_election.cc:302] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, 290f0ca7161744a19fb563f8db1551ae, a8d9ab7146de4910b532b52d842b5777; no voters:
I20260928 10:50:42.326803 8290 raft_consensus.cc:2781] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:42.327020 8290 raft_consensus.cc:482] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:42.327175 8290 raft_consensus.cc:3037] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.330201 8290 raft_consensus.cc:504] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.331054 8290 leader_election.cc:288] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 election: Requested vote from peers 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:42.331614 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea415151cdb24c1f8745406f434dfbcf" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f578a3fd4e48937a99d600405061"
I20260928 10:50:42.331642 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea415151cdb24c1f8745406f434dfbcf" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777"
I20260928 10:50:42.331907 8074 raft_consensus.cc:3037] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.331943 8214 raft_consensus.cc:3037] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.334707 8074 raft_consensus.cc:2454] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 1.
I20260928 10:50:42.334903 8214 raft_consensus.cc:2454] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 1.
I20260928 10:50:42.335374 7865 leader_election.cc:302] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, 290f0ca7161744a19fb563f8db1551ae; no voters:
I20260928 10:50:42.335781 8290 raft_consensus.cc:2781] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:42.336475 8290 raft_consensus.cc:686] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 1 LEADER]: Becoming Leader. State: Replica: 290f0ca7161744a19fb563f8db1551ae, State: Running, Role: LEADER
I20260928 10:50:42.337102 8290 consensus_queue.cc:237] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.343887 7776 catalog_manager.cc:5495] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 290f0ca7161744a19fb563f8db1551ae (127.5.227.129). New cstate: current_term: 1 leader_uuid: "290f0ca7161744a19fb563f8db1551ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } health_report { overall_health: UNKNOWN } } }
Sep 28 10:50:42 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592636, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:42.415491 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.398974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:46385 (local address 127.0.0.1:55076)
0928 10:50:42.399996 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.400015 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.400161 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.400730 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.400734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.401516 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:42.401853 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.401864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.402998 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.403006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.404094 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.404106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.404914 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.404932 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.405099 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.405866 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:42.405891 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:42.408750 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:42.412474 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:42.412493 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:42.412502 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:42.412970 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:42.413486 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:42.413490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:42.413493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:42.413643 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:42.414536 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:42.414544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:42.414737 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.414879 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.415112 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":668,"threads_started":1}
I20260928 10:50:42.415964 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.398883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.5.227.190:46385)
0928 10:50:42.399293 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.399299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.400137 (+ 838us) server_negotiation.cc:407] Connection header received
0928 10:50:42.400357 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.400362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.400431 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.400527 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:42.402057 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.402843 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.404262 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.404693 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.404866 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.409008 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:42.409013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:42.409017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:42.409079 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:42.412252 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.413168 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.413173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.413175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.413296 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:42.413816 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:42.413820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:42.413823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:42.414177 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:42.414311 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.415086 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.415402 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 10:50:42.422749 7776 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:55076:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:50:42.424427 7776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 10:50:42.428793 7985 tablet.cc:2369] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:50:42.434005 7914 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20bf84f82c4d36b04db40701d4e047 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e3f193c9c3d4150ae333f925a3dd7d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:50:42.434197 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20bf84f82c4d36b04db40701d4e047 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e3f193c9c3d4150ae333f925a3dd7d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:50:42.435482 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20bf84f82c4d36b04db40701d4e047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.435649 7914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20bf84f82c4d36b04db40701d4e047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.434978 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20bf84f82c4d36b04db40701d4e047 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e3f193c9c3d4150ae333f925a3dd7d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:50:42.436146 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20bf84f82c4d36b04db40701d4e047. 1 dirs total, 0 dirs full, 0 dirs failed
W20260928 10:50:42.438246 8125 tablet.cc:2369] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:50:42.443411 8284 tablet_bootstrap.cc:492] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: Bootstrap starting.
I20260928 10:50:42.445420 8284 tablet_bootstrap.cc:654] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.447930 8286 tablet_bootstrap.cc:492] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: Bootstrap starting.
I20260928 10:50:42.449235 8283 tablet_bootstrap.cc:492] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: Bootstrap starting.
I20260928 10:50:42.449838 8286 tablet_bootstrap.cc:654] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.450234 8284 tablet_bootstrap.cc:492] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: No bootstrap required, opened a new log
I20260928 10:50:42.450403 8284 ts_tablet_manager.cc:1397] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 10:50:42.451694 8283 tablet_bootstrap.cc:654] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.452039 8284 raft_consensus.cc:348] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.452548 8284 raft_consensus.cc:374] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.452673 8284 raft_consensus.cc:729] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.452826 8286 tablet_bootstrap.cc:492] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: No bootstrap required, opened a new log
I20260928 10:50:42.453111 8286 ts_tablet_manager.cc:1397] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 10:50:42.453022 8284 consensus_queue.cc:260] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.453925 8284 ts_tablet_manager.cc:1428] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 10:50:42.454423 8289 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.454831 8289 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:42.455497 8283 tablet_bootstrap.cc:492] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: No bootstrap required, opened a new log
I20260928 10:50:42.455714 8283 ts_tablet_manager.cc:1397] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 10:50:42.455618 8286 raft_consensus.cc:348] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.455935 8286 raft_consensus.cc:374] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.456043 8286 raft_consensus.cc:729] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ab7146de4910b532b52d842b5777, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.456293 8286 consensus_queue.cc:260] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.456971 8286 ts_tablet_manager.cc:1428] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 10:50:42.457331 8283 raft_consensus.cc:348] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.457123 8291 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.457581 8283 raft_consensus.cc:374] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.457647 8291 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:42.457670 8283 raft_consensus.cc:729] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f0ca7161744a19fb563f8db1551ae, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.457952 8283 consensus_queue.cc:260] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.458581 8283 ts_tablet_manager.cc:1428] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260928 10:50:42.458401 8290 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.458765 8290 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:42.545964 8289 raft_consensus.cc:482] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:42.546267 8289 raft_consensus.cc:504] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.548152 8289 leader_election.cc:288] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:42.558274 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.548680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:54901 (local address 127.5.227.129:43401)
0928 10:50:42.548957 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.548967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.549872 (+ 905us) server_negotiation.cc:407] Connection header received
0928 10:50:42.550068 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.550075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.550158 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.550277 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:42.551072 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.552087 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.555410 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.556935 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.557160 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.557265 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.557850 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.558012 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 10:50:42.558274 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.548729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:34343 (local address 127.5.227.131:34547)
0928 10:50:42.549033 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.549041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.549977 (+ 936us) server_negotiation.cc:407] Connection header received
0928 10:50:42.550175 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.550179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.550253 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.550345 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:42.551169 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.552175 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.555660 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.557160 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.557356 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.557411 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.557850 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.558043 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:50:42.558756 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.548441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:43401 (local address 127.5.227.130:54901)
0928 10:50:42.549714 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.549731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.549881 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.550290 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.550295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.550319 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:42.550794 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.550814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.552284 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.552291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.555132 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.555153 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.557256 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.557265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.557531 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.557632 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.557764 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.557954 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"mutex_wait_us":322,"thread_start_us":1037,"threads_started":1}
I20260928 10:50:42.559077 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.548550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:34547 (local address 127.5.227.130:34343)
0928 10:50:42.549823 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.549850 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.550026 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.550494 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.550500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.550523 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:42.551001 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.551018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.552305 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.552308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.555132 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.555156 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.557425 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.557431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.557640 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.557665 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.557764 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.557954 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":213,"thread_start_us":643,"threads_started":1}
I20260928 10:50:42.559859 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20bf84f82c4d36b04db40701d4e047" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777" is_pre_election: true
I20260928 10:50:42.560436 8214 raft_consensus.cc:2454] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 0.
I20260928 10:50:42.560474 7939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20bf84f82c4d36b04db40701d4e047" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae" is_pre_election: true
I20260928 10:50:42.561007 7939 raft_consensus.cc:2454] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 0.
I20260928 10:50:42.561111 8004 leader_election.cc:302] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, a8d9ab7146de4910b532b52d842b5777; no voters:
I20260928 10:50:42.561609 8289 raft_consensus.cc:2781] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:42.561729 8289 raft_consensus.cc:482] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:42.561879 8289 raft_consensus.cc:3037] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.564248 8289 raft_consensus.cc:504] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.564929 8289 leader_election.cc:288] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 election: Requested vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:42.565559 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20bf84f82c4d36b04db40701d4e047" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777"
I20260928 10:50:42.565572 7939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20bf84f82c4d36b04db40701d4e047" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae"
I20260928 10:50:42.565809 7939 raft_consensus.cc:3037] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.565833 8214 raft_consensus.cc:3037] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.568158 8214 raft_consensus.cc:2454] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 1.
I20260928 10:50:42.568413 7939 raft_consensus.cc:2454] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 1.
I20260928 10:50:42.568727 8004 leader_election.cc:302] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, a8d9ab7146de4910b532b52d842b5777; no voters:
I20260928 10:50:42.569285 8289 raft_consensus.cc:2781] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:42.569857 8289 raft_consensus.cc:686] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [term 1 LEADER]: Becoming Leader. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Running, Role: LEADER
I20260928 10:50:42.570505 8289 consensus_queue.cc:237] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.574791 8307 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: TxnStatusTablet state changed. Reason: New leader 2813f578a3fd4e48937a99d600405061. Latest consensus state: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.575281 8307 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:50:42.576498 8309 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:50:42.576834 8309 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:50:42.577171 7775 catalog_manager.cc:5495] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2813f578a3fd4e48937a99d600405061 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20260928 10:50:42.623147 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.617138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:46479 (local address 127.0.0.1:43200)
0928 10:50:42.617402 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.617423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.617618 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.618102 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.618106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.618118 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:50:42.618330 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.618339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.619482 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.619491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.620598 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.620610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.621378 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.621384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.621552 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.621624 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:50:42.622547 (+ 923us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:50:42.622574 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.622667 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.622836 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 10:50:42.623157 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.617165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.5.227.130:46479)
0928 10:50:42.617398 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.617405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.617516 (+ 111us) server_negotiation.cc:407] Connection header received
0928 10:50:42.617838 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.617843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.617933 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.618084 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:42.618529 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.619334 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.620751 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.621197 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.621358 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.621835 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:42.622429 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:42.622592 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.622699 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.622793 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260928 10:50:42.631935 8214 raft_consensus.cc:1261] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Refusing update from remote peer 2813f578a3fd4e48937a99d600405061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:42.631906 7939 raft_consensus.cc:1261] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Refusing update from remote peer 2813f578a3fd4e48937a99d600405061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:42.633036 8307 consensus_queue.cc:1035] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:42.633404 8289 consensus_queue.cc:1035] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:42.645617 8290 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: TxnStatusTablet state changed. Reason: New leader 2813f578a3fd4e48937a99d600405061. Latest consensus state: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.645958 8290 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:42.650970 8307 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.652256 8307 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:50:42.652946 8291 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: TxnStatusTablet state changed. Reason: New leader 2813f578a3fd4e48937a99d600405061. Latest consensus state: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.653239 8291 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:42.657274 8290 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.657599 8290 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P 290f0ca7161744a19fb563f8db1551ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:42.667344 8307 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.667665 8307 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P 2813f578a3fd4e48937a99d600405061: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:50:42.671864 8291 tablet_replica.cc:439] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } } }
I20260928 10:50:42.672154 8291 tablet_replica.cc:442] T 5e20bf84f82c4d36b04db40701d4e047 P a8d9ab7146de4910b532b52d842b5777: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:50:42.672313 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.664413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.5.227.129:43401)
0928 10:50:42.664625 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.664633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.664762 (+ 129us) server_negotiation.cc:407] Connection header received
0928 10:50:42.665082 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.665087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.665165 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.665326 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:42.665798 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.666570 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.668511 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.669131 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.669229 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.669864 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:42.670535 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:42.670712 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.671854 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.671946 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:50:42.692631 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.686157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:49893 (local address 127.5.227.130:46479)
0928 10:50:42.686390 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.686397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.686732 (+ 335us) server_negotiation.cc:407] Connection header received
0928 10:50:42.686961 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.686969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.687041 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.687148 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:42.687659 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.689046 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.690658 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.691136 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.691279 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.691811 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:42.691948 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:42.692057 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.692202 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.692284 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:50:42.692734 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.685918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:46479 (local address 127.5.227.129:49893)
0928 10:50:42.686613 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.686629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.686817 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.687161 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.687166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.687177 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:50:42.687494 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.687503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.689218 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.689226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.690495 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.690509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.691346 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.691353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.691489 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.691558 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:50:42.692060 (+ 502us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:50:42.692074 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.692181 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.692330 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":346,"threads_started":1}
I20260928 10:50:42.706821 8214 raft_consensus.cc:1261] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Refusing update from remote peer 290f0ca7161744a19fb563f8db1551ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:42.707201 8074 raft_consensus.cc:1261] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Refusing update from remote peer 290f0ca7161744a19fb563f8db1551ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:42.707993 8294 consensus_queue.cc:1035] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:42.708827 8290 consensus_queue.cc:1035] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:42.715451 8313 mvcc.cc:205] Tried to move back new op lower bound from 7334267464519933952 to 7334267463018000384. Current Snapshot: MvccSnapshot[applied={T|T < 7334267464519933952}]
I20260928 10:50:42.721266 8314 mvcc.cc:205] Tried to move back new op lower bound from 7334267464519933952 to 7334267463018000384. Current Snapshot: MvccSnapshot[applied={T|T < 7334267464519933952}]
I20260928 10:50:42.723078 8312 mvcc.cc:205] Tried to move back new op lower bound from 7334267464519933952 to 7334267463018000384. Current Snapshot: MvccSnapshot[applied={T|T < 7334267464519933952}]
I20260928 10:50:42.755195 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.746479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:46433 (local address 127.5.227.129:43401)
0928 10:50:42.746735 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:42.746746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:42.747446 (+ 700us) server_negotiation.cc:407] Connection header received
0928 10:50:42.747645 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:42.747651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:42.747765 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:42.747861 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:42.748868 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.749810 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.752184 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.752875 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.753044 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.754170 (+ 1126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:42.754513 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:42.754631 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:42.754789 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 10:50:42.754872 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:50:42.755575 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:42.746374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:43401 (local address 127.5.227.130:46433)
0928 10:50:42.747293 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:42.747310 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:42.747504 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:42.748102 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:42.748108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:42.748123 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:50:42.748599 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.748611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.749956 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.749962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.751987 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:42.752007 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:42.753529 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:42.753538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:42.753767 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:42.753859 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:50:42.754656 (+ 797us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:50:42.754671 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 10:50:42.754755 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:50:42.754918 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":558,"threads_started":1}
I20260928 10:50:42.806653 7777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:42.806955 7777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:42.811019 7777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Sending DeleteTablet for 3 replicas of tablet ea415151cdb24c1f8745406f434dfbcf
I20260928 10:50:42.812273 7914 tablet_service.cc:1514] Processing DeleteTablet for tablet ea415151cdb24c1f8745406f434dfbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:42 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:32954
I20260928 10:50:42.812273 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet ea415151cdb24c1f8745406f434dfbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:42 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:43190
I20260928 10:50:42.813603 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet ea415151cdb24c1f8745406f434dfbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:42 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:59928
I20260928 10:50:42.818322 7777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:50:42.819077 7777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:42.827013 7913 tablet_service.cc:1467] Processing CreateTablet for tablet d9275f84bffb4489b92719222dfa0962 (DEFAULT_TABLE table=test-table [id=dd7e45b4d7954159a05ab83df96283af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:42.826990 8193 tablet_service.cc:1467] Processing CreateTablet for tablet d9275f84bffb4489b92719222dfa0962 (DEFAULT_TABLE table=test-table [id=dd7e45b4d7954159a05ab83df96283af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:42.827584 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9275f84bffb4489b92719222dfa0962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.827646 8193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9275f84bffb4489b92719222dfa0962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.834126 8333 tablet_replica.cc:330] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: stopping tablet replica
I20260928 10:50:42.834641 8333 raft_consensus.cc:2229] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:50:42.835307 8333 raft_consensus.cc:2258] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:42.835959 8286 tablet_bootstrap.cc:492] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: Bootstrap starting.
I20260928 10:50:42.836346 8053 tablet_service.cc:1467] Processing CreateTablet for tablet d9275f84bffb4489b92719222dfa0962 (DEFAULT_TABLE table=test-table [id=dd7e45b4d7954159a05ab83df96283af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:42.836936 8053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9275f84bffb4489b92719222dfa0962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:42.838018 8286 tablet_bootstrap.cc:654] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.838110 8333 ts_tablet_manager.cc:1905] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:42.843317 8283 tablet_bootstrap.cc:492] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: Bootstrap starting.
I20260928 10:50:42.843394 8284 tablet_bootstrap.cc:492] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: Bootstrap starting.
I20260928 10:50:42.844094 8335 tablet_replica.cc:330] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: stopping tablet replica
I20260928 10:50:42.844586 8335 raft_consensus.cc:2229] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:42.845113 8284 tablet_bootstrap.cc:654] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.845273 8283 tablet_bootstrap.cc:654] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:42.845701 8335 raft_consensus.cc:2258] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:42.847477 8333 ts_tablet_manager.cc:1918] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:42.847692 8333 log.cc:1196] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/ea415151cdb24c1f8745406f434dfbcf
I20260928 10:50:42.848296 8333 ts_tablet_manager.cc:1939] T ea415151cdb24c1f8745406f434dfbcf P 290f0ca7161744a19fb563f8db1551ae: Deleting consensus metadata
I20260928 10:50:42.850550 7764 catalog_manager.cc:4841] TS 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401): tablet ea415151cdb24c1f8745406f434dfbcf (table test-table [id=aee1f2bf9d6240efa91a81e29cad798a]) successfully deleted
I20260928 10:50:42.851048 8334 tablet_replica.cc:330] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: stopping tablet replica
I20260928 10:50:42.851558 8334 raft_consensus.cc:2229] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:42.852042 8334 raft_consensus.cc:2258] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:42.852538 8335 ts_tablet_manager.cc:1905] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:42.857684 8334 ts_tablet_manager.cc:1905] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:42.866709 8283 tablet_bootstrap.cc:492] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: No bootstrap required, opened a new log
I20260928 10:50:42.867002 8283 ts_tablet_manager.cc:1397] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.003s
I20260928 10:50:42.867913 8283 raft_consensus.cc:348] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.868314 8283 raft_consensus.cc:374] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.868446 8283 raft_consensus.cc:729] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f0ca7161744a19fb563f8db1551ae, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.868839 8283 consensus_queue.cc:260] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.869587 8283 ts_tablet_manager.cc:1428] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:50:42.871608 8284 tablet_bootstrap.cc:492] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: No bootstrap required, opened a new log
I20260928 10:50:42.871914 8284 ts_tablet_manager.cc:1397] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.012s
I20260928 10:50:42.872864 8284 raft_consensus.cc:348] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.873193 8284 raft_consensus.cc:374] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.873337 8284 raft_consensus.cc:729] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.873709 8284 consensus_queue.cc:260] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.874677 8286 tablet_bootstrap.cc:492] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: No bootstrap required, opened a new log
I20260928 10:50:42.874866 8286 ts_tablet_manager.cc:1397] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.007s
I20260928 10:50:42.875238 8335 ts_tablet_manager.cc:1918] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:42.875423 8335 log.cc:1196] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/ea415151cdb24c1f8745406f434dfbcf
I20260928 10:50:42.875836 8286 raft_consensus.cc:348] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.876013 8335 ts_tablet_manager.cc:1939] T ea415151cdb24c1f8745406f434dfbcf P a8d9ab7146de4910b532b52d842b5777: Deleting consensus metadata
I20260928 10:50:42.876121 8286 raft_consensus.cc:374] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:42.876222 8286 raft_consensus.cc:729] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ab7146de4910b532b52d842b5777, State: Initialized, Role: FOLLOWER
I20260928 10:50:42.876585 8286 consensus_queue.cc:260] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.877218 8284 ts_tablet_manager.cc:1428] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260928 10:50:42.878108 8286 ts_tablet_manager.cc:1428] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 10:50:42.879524 7761 catalog_manager.cc:4841] TS a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547): tablet ea415151cdb24c1f8745406f434dfbcf (table test-table [id=aee1f2bf9d6240efa91a81e29cad798a]) successfully deleted
I20260928 10:50:42.879621 8334 ts_tablet_manager.cc:1918] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:42.879822 8334 log.cc:1196] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/ea415151cdb24c1f8745406f434dfbcf
I20260928 10:50:42.880498 8334 ts_tablet_manager.cc:1939] T ea415151cdb24c1f8745406f434dfbcf P 2813f578a3fd4e48937a99d600405061: Deleting consensus metadata
I20260928 10:50:42.882377 7762 catalog_manager.cc:4841] TS 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479): tablet ea415151cdb24c1f8745406f434dfbcf (table test-table [id=aee1f2bf9d6240efa91a81e29cad798a]) successfully deleted
I20260928 10:50:42.956439 8330 raft_consensus.cc:482] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:42.956717 8330 raft_consensus.cc:504] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.957641 8330 leader_election.cc:288] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:42.958279 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9275f84bffb4489b92719222dfa0962" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f578a3fd4e48937a99d600405061" is_pre_election: true
I20260928 10:50:42.958425 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9275f84bffb4489b92719222dfa0962" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777" is_pre_election: true
I20260928 10:50:42.958585 8074 raft_consensus.cc:2454] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 0.
I20260928 10:50:42.958712 8213 raft_consensus.cc:2454] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 0.
I20260928 10:50:42.959458 7864 leader_election.cc:302] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290f0ca7161744a19fb563f8db1551ae, a8d9ab7146de4910b532b52d842b5777; no voters:
I20260928 10:50:42.959872 8330 raft_consensus.cc:2781] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:42.960000 8330 raft_consensus.cc:482] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:42.960096 8330 raft_consensus.cc:3037] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.963492 8330 raft_consensus.cc:504] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.964497 8330 leader_election.cc:288] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 election: Requested vote from peers 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:42.964974 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9275f84bffb4489b92719222dfa0962" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777"
I20260928 10:50:42.965130 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9275f84bffb4489b92719222dfa0962" candidate_uuid: "290f0ca7161744a19fb563f8db1551ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f578a3fd4e48937a99d600405061"
I20260928 10:50:42.965221 8213 raft_consensus.cc:3037] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.965344 8074 raft_consensus.cc:3037] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:42.968649 8213 raft_consensus.cc:2454] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 1.
I20260928 10:50:42.968989 8074 raft_consensus.cc:2454] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290f0ca7161744a19fb563f8db1551ae in term 1.
I20260928 10:50:42.969323 7864 leader_election.cc:302] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290f0ca7161744a19fb563f8db1551ae, a8d9ab7146de4910b532b52d842b5777; no voters:
I20260928 10:50:42.970432 8330 raft_consensus.cc:2781] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:42.970990 8330 raft_consensus.cc:686] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 1 LEADER]: Becoming Leader. State: Replica: 290f0ca7161744a19fb563f8db1551ae, State: Running, Role: LEADER
I20260928 10:50:42.971333 8330 consensus_queue.cc:237] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:42.978199 7775 catalog_manager.cc:5495] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 290f0ca7161744a19fb563f8db1551ae (127.5.227.129). New cstate: current_term: 1 leader_uuid: "290f0ca7161744a19fb563f8db1551ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20260928 10:50:43.034188 8074 raft_consensus.cc:1261] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Refusing update from remote peer 290f0ca7161744a19fb563f8db1551ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:43.034998 8294 consensus_queue.cc:1035] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:43.035400 8214 raft_consensus.cc:1261] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Refusing update from remote peer 290f0ca7161744a19fb563f8db1551ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:43.036106 8294 consensus_queue.cc:1035] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:50:43.090533 8313 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 10:50:44.706445 7775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:44.706702 7775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:44.711071 7775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Sending DeleteTablet for 3 replicas of tablet d9275f84bffb4489b92719222dfa0962
I20260928 10:50:44.712028 7913 tablet_service.cc:1514] Processing DeleteTablet for tablet d9275f84bffb4489b92719222dfa0962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:44 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:32954
I20260928 10:50:44.712038 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet d9275f84bffb4489b92719222dfa0962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:44 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:43190
I20260928 10:50:44.712093 8193 tablet_service.cc:1514] Processing DeleteTablet for tablet d9275f84bffb4489b92719222dfa0962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:44 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:59928
I20260928 10:50:44.725204 8366 tablet_replica.cc:330] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: stopping tablet replica
I20260928 10:50:44.725528 8366 raft_consensus.cc:2229] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:50:44.726018 8366 raft_consensus.cc:2258] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:44.727494 8366 ts_tablet_manager.cc:1905] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:44.732182 8367 tablet_replica.cc:330] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: stopping tablet replica
I20260928 10:50:44.732457 8368 tablet_replica.cc:330] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: stopping tablet replica
I20260928 10:50:44.732481 8367 raft_consensus.cc:2229] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:44.732692 8368 raft_consensus.cc:2229] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:44.732736 8367 raft_consensus.cc:2258] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:44.733036 8368 raft_consensus.cc:2258] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:44.734252 8368 ts_tablet_manager.cc:1905] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:44.734984 8367 ts_tablet_manager.cc:1905] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:44.736265 8366 ts_tablet_manager.cc:1918] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:50:44.736405 8366 log.cc:1196] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/d9275f84bffb4489b92719222dfa0962
I20260928 10:50:44.736836 8366 ts_tablet_manager.cc:1939] T d9275f84bffb4489b92719222dfa0962 P 290f0ca7161744a19fb563f8db1551ae: Deleting consensus metadata
I20260928 10:50:44.738157 7764 catalog_manager.cc:4841] TS 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401): tablet d9275f84bffb4489b92719222dfa0962 (table test-table [id=dd7e45b4d7954159a05ab83df96283af]) successfully deleted
I20260928 10:50:44.741350 8367 ts_tablet_manager.cc:1918] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:50:44.741490 8367 log.cc:1196] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/d9275f84bffb4489b92719222dfa0962
I20260928 10:50:44.741925 8367 ts_tablet_manager.cc:1939] T d9275f84bffb4489b92719222dfa0962 P 2813f578a3fd4e48937a99d600405061: Deleting consensus metadata
I20260928 10:50:44.742216 8368 ts_tablet_manager.cc:1918] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:50:44.742491 8368 log.cc:1196] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/d9275f84bffb4489b92719222dfa0962
I20260928 10:50:44.743010 8368 ts_tablet_manager.cc:1939] T d9275f84bffb4489b92719222dfa0962 P a8d9ab7146de4910b532b52d842b5777: Deleting consensus metadata
I20260928 10:50:44.743085 7762 catalog_manager.cc:4841] TS 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479): tablet d9275f84bffb4489b92719222dfa0962 (table test-table [id=dd7e45b4d7954159a05ab83df96283af]) successfully deleted
I20260928 10:50:44.744341 7761 catalog_manager.cc:4841] TS a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547): tablet d9275f84bffb4489b92719222dfa0962 (table test-table [id=dd7e45b4d7954159a05ab83df96283af]) successfully deleted
Sep 28 10:50:44 dist-test-slave-8w8g krb5kdc[7732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592644, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:50:44 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592644, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:44.774261 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:44.759216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.5.227.190:46385)
0928 10:50:44.759880 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:44.759889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:44.759964 (+ 75us) server_negotiation.cc:407] Connection header received
0928 10:50:44.760248 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:44.760253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:44.760315 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:44.760458 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:44.762228 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.763000 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.764318 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.764725 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.764862 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:44.768860 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:44.768866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:44.768869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:44.768906 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:44.771544 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:44.772288 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:44.772298 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:44.772301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:44.772455 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:44.772855 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:44.772859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:44.772861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:44.773145 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:44.773389 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:44.773692 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 10:50:44.773912 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":336,"threads_started":1}
I20260928 10:50:44.778064 7775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:50:44.778738 7775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:44.785593 7913 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe2ab536f54e36941b6b1bdd26064a (DEFAULT_TABLE table=test-table [id=360cc109b4614e6ebcee0bd61e348c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:44.785821 8054 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe2ab536f54e36941b6b1bdd26064a (DEFAULT_TABLE table=test-table [id=360cc109b4614e6ebcee0bd61e348c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:44.786108 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe2ab536f54e36941b6b1bdd26064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:44.786023 8193 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe2ab536f54e36941b6b1bdd26064a (DEFAULT_TABLE table=test-table [id=360cc109b4614e6ebcee0bd61e348c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:44.786245 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe2ab536f54e36941b6b1bdd26064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:44.786463 8193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe2ab536f54e36941b6b1bdd26064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:44.791038 8377 tablet_bootstrap.cc:492] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: Bootstrap starting.
I20260928 10:50:44.792909 8377 tablet_bootstrap.cc:654] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:44.793586 8378 tablet_bootstrap.cc:492] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: Bootstrap starting.
I20260928 10:50:44.794194 8379 tablet_bootstrap.cc:492] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: Bootstrap starting.
I20260928 10:50:44.795444 8377 tablet_bootstrap.cc:492] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: No bootstrap required, opened a new log
I20260928 10:50:44.795647 8377 ts_tablet_manager.cc:1397] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 10:50:44.796046 8379 tablet_bootstrap.cc:654] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:44.796558 8378 tablet_bootstrap.cc:654] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:44.796720 8377 raft_consensus.cc:348] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.796968 8377 raft_consensus.cc:374] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:44.797096 8377 raft_consensus.cc:729] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f0ca7161744a19fb563f8db1551ae, State: Initialized, Role: FOLLOWER
I20260928 10:50:44.797442 8377 consensus_queue.cc:260] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.798599 8377 ts_tablet_manager.cc:1428] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 10:50:44.798822 8378 tablet_bootstrap.cc:492] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: No bootstrap required, opened a new log
I20260928 10:50:44.798961 8378 ts_tablet_manager.cc:1397] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 10:50:44.799690 8379 tablet_bootstrap.cc:492] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: No bootstrap required, opened a new log
I20260928 10:50:44.799872 8379 ts_tablet_manager.cc:1397] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 10:50:44.799685 8378 raft_consensus.cc:348] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.800005 8378 raft_consensus.cc:374] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:44.800104 8378 raft_consensus.cc:729] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Initialized, Role: FOLLOWER
I20260928 10:50:44.800350 8378 consensus_queue.cc:260] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.800705 8379 raft_consensus.cc:348] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.800884 8378 ts_tablet_manager.cc:1428] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 10:50:44.800923 8379 raft_consensus.cc:374] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:44.801050 8379 raft_consensus.cc:729] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ab7146de4910b532b52d842b5777, State: Initialized, Role: FOLLOWER
I20260928 10:50:44.801370 8379 consensus_queue.cc:260] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.802284 8379 ts_tablet_manager.cc:1428] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:50:44.973716 8383 raft_consensus.cc:482] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:44.974005 8383 raft_consensus.cc:504] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.975567 8383 leader_election.cc:288] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479)
I20260928 10:50:44.986145 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:44.976029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:51535 (local address 127.5.227.129:43401)
0928 10:50:44.977828 (+ 1799us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:44.977833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:44.977853 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:50:44.977913 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:44.977917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:44.977984 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:44.978086 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:44.979007 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.980273 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.983400 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.984700 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.984946 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:44.985025 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:44.985656 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 10:50:44.985817 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":1525,"threads_started":1}
I20260928 10:50:44.986150 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:44.975999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:42585 (local address 127.5.227.130:46479)
0928 10:50:44.977099 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:44.977109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:44.977778 (+ 669us) server_negotiation.cc:407] Connection header received
0928 10:50:44.977870 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:44.977874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:44.977953 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:44.978075 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:50:44.979002 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.980264 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.983398 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.984692 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.984944 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:44.985025 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:44.985656 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 10:50:44.985848 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":766,"threads_started":1}
I20260928 10:50:44.986670 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:44.976408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:46479 (local address 127.5.227.131:42585)
0928 10:50:44.977637 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:44.977660 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:44.977768 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:44.978123 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:44.978128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:44.978162 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:44.978751 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:44.978760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.980491 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.980494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:44.983175 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:44.983194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.985089 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.985094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:44.985436 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:44.985461 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:50:44.985546 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:50:44.985740 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":325,"spinlock_wait_cycles":1408,"thread_start_us":896,"threads_started":1}
I20260928 10:50:44.986666 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:44.975978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:43401 (local address 127.5.227.131:51535)
0928 10:50:44.976775 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:44.976811 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:44.977009 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:44.978123 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:44.978128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:44.978167 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:50:44.978669 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:44.978688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.980484 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.980490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:44.983175 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:44.983194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:44.985065 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:44.985078 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:44.985308 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:44.985428 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 10:50:44.985546 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 10:50:44.985767 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":333,"spinlock_wait_cycles":2944,"thread_start_us":401,"threads_started":1}
I20260928 10:50:44.987061 8347 raft_consensus.cc:482] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:44.987900 7939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "a8d9ab7146de4910b532b52d842b5777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae" is_pre_election: true
I20260928 10:50:44.987309 8347 raft_consensus.cc:504] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.988036 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "a8d9ab7146de4910b532b52d842b5777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f578a3fd4e48937a99d600405061" is_pre_election: true
I20260928 10:50:44.988186 7939 raft_consensus.cc:2454] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d9ab7146de4910b532b52d842b5777 in term 0.
I20260928 10:50:44.988262 8074 raft_consensus.cc:2454] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d9ab7146de4910b532b52d842b5777 in term 0.
I20260928 10:50:44.988634 8347 leader_election.cc:288] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:44.988831 8145 leader_election.cc:302] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, a8d9ab7146de4910b532b52d842b5777; no voters:
I20260928 10:50:44.989257 8383 raft_consensus.cc:2781] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:44.989392 8383 raft_consensus.cc:482] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:44.989487 8383 raft_consensus.cc:3037] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:44.989508 7939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae" is_pre_election: true
I20260928 10:50:44.989807 7939 raft_consensus.cc:2454] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 0.
I20260928 10:50:44.989941 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777" is_pre_election: true
I20260928 10:50:44.990284 8007 leader_election.cc:302] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, 290f0ca7161744a19fb563f8db1551ae; no voters:
I20260928 10:50:44.990577 8347 raft_consensus.cc:2781] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:44.990685 8347 raft_consensus.cc:482] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:44.990761 8347 raft_consensus.cc:3037] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:44.992904 8383 raft_consensus.cc:504] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.993379 8214 raft_consensus.cc:2379] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2813f578a3fd4e48937a99d600405061 in current term 1: Already voted for candidate a8d9ab7146de4910b532b52d842b5777 in this term.
I20260928 10:50:44.993948 8347 raft_consensus.cc:504] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:44.994558 8383 leader_election.cc:288] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [CANDIDATE]: Term 1 election: Requested vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479)
I20260928 10:50:44.995119 7938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "a8d9ab7146de4910b532b52d842b5777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae"
I20260928 10:50:44.995155 7939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae"
I20260928 10:50:44.995216 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "a8d9ab7146de4910b532b52d842b5777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f578a3fd4e48937a99d600405061"
I20260928 10:50:44.995419 7939 raft_consensus.cc:3037] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:44.995510 8074 raft_consensus.cc:2379] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a8d9ab7146de4910b532b52d842b5777 in current term 1: Already voted for candidate 2813f578a3fd4e48937a99d600405061 in this term.
I20260928 10:50:44.995539 8347 leader_election.cc:288] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 election: Requested vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:44.996173 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe2ab536f54e36941b6b1bdd26064a" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777"
I20260928 10:50:44.996452 8147 leader_election.cc:302] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8d9ab7146de4910b532b52d842b5777; no voters: 2813f578a3fd4e48937a99d600405061, 290f0ca7161744a19fb563f8db1551ae
I20260928 10:50:44.996534 8214 raft_consensus.cc:2379] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2813f578a3fd4e48937a99d600405061 in current term 1: Already voted for candidate a8d9ab7146de4910b532b52d842b5777 in this term.
I20260928 10:50:44.996860 8383 raft_consensus.cc:2726] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 10:50:44.998119 7939 raft_consensus.cc:2454] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 1.
I20260928 10:50:44.998530 8007 leader_election.cc:302] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, 290f0ca7161744a19fb563f8db1551ae; no voters: a8d9ab7146de4910b532b52d842b5777
I20260928 10:50:44.998912 8347 raft_consensus.cc:2781] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:44.999094 8347 raft_consensus.cc:686] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 1 LEADER]: Becoming Leader. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Running, Role: LEADER
I20260928 10:50:44.999468 8347 consensus_queue.cc:237] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.002619 7777 catalog_manager.cc:5495] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2813f578a3fd4e48937a99d600405061 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20260928 10:50:45.057549 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:45.051255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.5.227.130:46479)
0928 10:50:45.051783 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:45.051787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:45.051804 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:50:45.051876 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:45.051879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:45.051940 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:45.052024 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:45.052474 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:45.053351 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:45.055331 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:45.056024 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:45.056183 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:45.056736 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:45.056860 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:45.056944 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:45.057126 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 10:50:45.057194 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260928 10:50:45.073098 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:45.067397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:57675 (local address 127.5.227.130:46479)
0928 10:50:45.067571 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:45.067579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:45.068173 (+ 594us) server_negotiation.cc:407] Connection header received
0928 10:50:45.068364 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:45.068392 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:45.068448 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:45.068540 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:45.068951 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:45.069701 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:45.071055 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:45.071461 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:45.071598 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:45.072063 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:45.072269 (+ 206us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:45.072387 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:45.072563 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 10:50:45.072635 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:50:45.073223 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:45.067449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:46479 (local address 127.5.227.130:57675)
0928 10:50:45.068062 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:45.068081 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:45.068242 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:45.068549 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:45.068553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:45.068563 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:50:45.068812 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:45.068820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:45.069827 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:45.069834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:45.070922 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:45.070933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:45.071635 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:45.071641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:45.071835 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:45.071877 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:50:45.072434 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:50:45.072446 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:50:45.072525 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 10:50:45.072631 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":360,"threads_started":1}
I20260928 10:50:45.081168 8214 raft_consensus.cc:1261] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Refusing update from remote peer 2813f578a3fd4e48937a99d600405061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:45.081468 7939 raft_consensus.cc:1261] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Refusing update from remote peer 2813f578a3fd4e48937a99d600405061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:45.081755 8352 consensus_queue.cc:1035] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:45.082131 8343 consensus_queue.cc:1035] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:45.132371 7777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:45.132647 7777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:45.135854 7777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Sending DeleteTablet for 3 replicas of tablet c9fe2ab536f54e36941b6b1bdd26064a
I20260928 10:50:45.136984 7913 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fe2ab536f54e36941b6b1bdd26064a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:45 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:32954
I20260928 10:50:45.136989 8053 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fe2ab536f54e36941b6b1bdd26064a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:45 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:43190
I20260928 10:50:45.137358 8367 tablet_replica.cc:330] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: stopping tablet replica
I20260928 10:50:45.137351 8193 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fe2ab536f54e36941b6b1bdd26064a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:45 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:59928
I20260928 10:50:45.137570 8367 raft_consensus.cc:2229] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:50:45.137703 8368 tablet_replica.cc:330] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: stopping tablet replica
I20260928 10:50:45.137866 8368 raft_consensus.cc:2229] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:45.138056 8368 raft_consensus.cc:2258] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:45.138089 8367 raft_consensus.cc:2258] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:45.138377 8366 tablet_replica.cc:330] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: stopping tablet replica
I20260928 10:50:45.138593 8366 raft_consensus.cc:2229] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:45.138492 7777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:50:45.139402 7777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:50:45.139403 8366 raft_consensus.cc:2258] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:45.140465 8368 ts_tablet_manager.cc:1905] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:45.140486 8367 ts_tablet_manager.cc:1905] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:45.140627 8366 ts_tablet_manager.cc:1905] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:45.147830 8367 ts_tablet_manager.cc:1918] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:45.148046 8367 log.cc:1196] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/c9fe2ab536f54e36941b6b1bdd26064a
I20260928 10:50:45.148056 8366 ts_tablet_manager.cc:1918] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:45.148228 8366 log.cc:1196] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/c9fe2ab536f54e36941b6b1bdd26064a
I20260928 10:50:45.148626 8367 ts_tablet_manager.cc:1939] T c9fe2ab536f54e36941b6b1bdd26064a P 2813f578a3fd4e48937a99d600405061: Deleting consensus metadata
I20260928 10:50:45.148645 8366 ts_tablet_manager.cc:1939] T c9fe2ab536f54e36941b6b1bdd26064a P 290f0ca7161744a19fb563f8db1551ae: Deleting consensus metadata
I20260928 10:50:45.149801 7764 catalog_manager.cc:4841] TS 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401): tablet c9fe2ab536f54e36941b6b1bdd26064a (table test-table [id=360cc109b4614e6ebcee0bd61e348c32]) successfully deleted
I20260928 10:50:45.150162 7762 catalog_manager.cc:4841] TS 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479): tablet c9fe2ab536f54e36941b6b1bdd26064a (table test-table [id=360cc109b4614e6ebcee0bd61e348c32]) successfully deleted
I20260928 10:50:45.150532 8368 ts_tablet_manager.cc:1918] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:50:45.150774 8368 log.cc:1196] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/c9fe2ab536f54e36941b6b1bdd26064a
I20260928 10:50:45.151209 8368 ts_tablet_manager.cc:1939] T c9fe2ab536f54e36941b6b1bdd26064a P a8d9ab7146de4910b532b52d842b5777: Deleting consensus metadata
I20260928 10:50:45.152511 7761 catalog_manager.cc:4841] TS a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547): tablet c9fe2ab536f54e36941b6b1bdd26064a (table test-table [id=360cc109b4614e6ebcee0bd61e348c32]) successfully deleted
I20260928 10:50:45.155467 7913 tablet_service.cc:1467] Processing CreateTablet for tablet 9163a1a3e18a4119b45f81ced8b0af56 (DEFAULT_TABLE table=test-table [id=8b86807113f74d08a62587704d064b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:45.155479 8053 tablet_service.cc:1467] Processing CreateTablet for tablet 9163a1a3e18a4119b45f81ced8b0af56 (DEFAULT_TABLE table=test-table [id=8b86807113f74d08a62587704d064b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:45.155931 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9163a1a3e18a4119b45f81ced8b0af56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:45.155958 8053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9163a1a3e18a4119b45f81ced8b0af56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:45.160202 8193 tablet_service.cc:1467] Processing CreateTablet for tablet 9163a1a3e18a4119b45f81ced8b0af56 (DEFAULT_TABLE table=test-table [id=8b86807113f74d08a62587704d064b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:50:45.160662 8193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9163a1a3e18a4119b45f81ced8b0af56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:45.163540 8378 tablet_bootstrap.cc:492] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: Bootstrap starting.
I20260928 10:50:45.165521 8378 tablet_bootstrap.cc:654] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:45.166862 8379 tablet_bootstrap.cc:492] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: Bootstrap starting.
I20260928 10:50:45.168457 8378 tablet_bootstrap.cc:492] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: No bootstrap required, opened a new log
I20260928 10:50:45.168772 8378 ts_tablet_manager.cc:1397] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 10:50:45.169181 8379 tablet_bootstrap.cc:654] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:45.172899 8377 tablet_bootstrap.cc:492] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: Bootstrap starting.
I20260928 10:50:45.174551 8379 tablet_bootstrap.cc:492] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: No bootstrap required, opened a new log
I20260928 10:50:45.174685 8377 tablet_bootstrap.cc:654] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:45.174739 8379 ts_tablet_manager.cc:1397] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 10:50:45.175566 8379 raft_consensus.cc:348] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.175942 8379 raft_consensus.cc:374] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:45.176044 8379 raft_consensus.cc:729] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ab7146de4910b532b52d842b5777, State: Initialized, Role: FOLLOWER
I20260928 10:50:45.176419 8379 consensus_queue.cc:260] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.177888 8378 raft_consensus.cc:348] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.180472 8379 ts_tablet_manager.cc:1428] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260928 10:50:45.180687 8378 raft_consensus.cc:374] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:45.180831 8378 raft_consensus.cc:729] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Initialized, Role: FOLLOWER
I20260928 10:50:45.181385 8378 consensus_queue.cc:260] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.182013 8378 ts_tablet_manager.cc:1428] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20260928 10:50:45.189047 8377 tablet_bootstrap.cc:492] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: No bootstrap required, opened a new log
I20260928 10:50:45.189333 8377 ts_tablet_manager.cc:1397] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20260928 10:50:45.190682 8377 raft_consensus.cc:348] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.190948 8377 raft_consensus.cc:374] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:45.191259 8377 raft_consensus.cc:729] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f0ca7161744a19fb563f8db1551ae, State: Initialized, Role: FOLLOWER
I20260928 10:50:45.191555 8377 consensus_queue.cc:260] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.192099 8377 ts_tablet_manager.cc:1428] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:50:45.275667 8343 raft_consensus.cc:482] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:50:45.276024 8343 raft_consensus.cc:504] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.276773 8343 leader_election.cc:288] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:45.277354 7939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163a1a3e18a4119b45f81ced8b0af56" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae" is_pre_election: true
I20260928 10:50:45.277360 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163a1a3e18a4119b45f81ced8b0af56" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777" is_pre_election: true
I20260928 10:50:45.277614 7939 raft_consensus.cc:2454] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 0.
I20260928 10:50:45.277637 8213 raft_consensus.cc:2454] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 0.
I20260928 10:50:45.278113 8007 leader_election.cc:302] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, 290f0ca7161744a19fb563f8db1551ae; no voters:
I20260928 10:50:45.278453 8343 raft_consensus.cc:2781] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:50:45.278566 8343 raft_consensus.cc:482] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:50:45.278674 8343 raft_consensus.cc:3037] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:45.281836 8343 raft_consensus.cc:504] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.282505 8343 leader_election.cc:288] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 election: Requested vote from peers 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401), a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547)
I20260928 10:50:45.283082 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163a1a3e18a4119b45f81ced8b0af56" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ab7146de4910b532b52d842b5777"
I20260928 10:50:45.283111 7939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163a1a3e18a4119b45f81ced8b0af56" candidate_uuid: "2813f578a3fd4e48937a99d600405061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f0ca7161744a19fb563f8db1551ae"
I20260928 10:50:45.283795 8213 raft_consensus.cc:3037] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:45.283830 7939 raft_consensus.cc:3037] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:45.286047 8213 raft_consensus.cc:2454] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 1.
I20260928 10:50:45.286552 8004 leader_election.cc:302] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f578a3fd4e48937a99d600405061, a8d9ab7146de4910b532b52d842b5777; no voters:
I20260928 10:50:45.286780 7939 raft_consensus.cc:2454] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2813f578a3fd4e48937a99d600405061 in term 1.
I20260928 10:50:45.286888 8343 raft_consensus.cc:2781] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:45.287247 8343 raft_consensus.cc:686] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 1 LEADER]: Becoming Leader. State: Replica: 2813f578a3fd4e48937a99d600405061, State: Running, Role: LEADER
I20260928 10:50:45.287518 8343 consensus_queue.cc:237] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } }
I20260928 10:50:45.290509 7775 catalog_manager.cc:5495] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2813f578a3fd4e48937a99d600405061 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "2813f578a3fd4e48937a99d600405061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2813f578a3fd4e48937a99d600405061" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 46479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20260928 10:50:45.320544 7938 raft_consensus.cc:1261] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Refusing update from remote peer 2813f578a3fd4e48937a99d600405061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:45.321349 8347 consensus_queue.cc:1035] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290f0ca7161744a19fb563f8db1551ae" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:50:45.321561 8213 raft_consensus.cc:1261] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Refusing update from remote peer 2813f578a3fd4e48937a99d600405061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:50:45.322795 8343 consensus_queue.cc:1035] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d9ab7146de4910b532b52d842b5777" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260928 10:50:45.352273 8345 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 10:50:46.947110 7775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:46.947448 7775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:50:46.951794 7775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe8fcbbfa2c047c8b65cfc447cd66cb2: Sending DeleteTablet for 3 replicas of tablet 9163a1a3e18a4119b45f81ced8b0af56
I20260928 10:50:46.952767 7913 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163a1a3e18a4119b45f81ced8b0af56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:46 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:32954
I20260928 10:50:46.952939 8053 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163a1a3e18a4119b45f81ced8b0af56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:46 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:43190
I20260928 10:50:46.952951 8193 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163a1a3e18a4119b45f81ced8b0af56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:50:46 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:59928
I20260928 10:50:46.954089 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7848
I20260928 10:50:46.956895 8425 tablet_replica.cc:330] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: stopping tablet replica
I20260928 10:50:46.957234 8425 raft_consensus.cc:2229] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:46.957584 8425 raft_consensus.cc:2258] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:46.959659 8425 ts_tablet_manager.cc:1905] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:46.969497 8425 ts_tablet_manager.cc:1918] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:50:46.970145 8426 tablet_replica.cc:330] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: stopping tablet replica
I20260928 10:50:46.970412 8425 log.cc:1196] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/9163a1a3e18a4119b45f81ced8b0af56
I20260928 10:50:46.970402 8426 raft_consensus.cc:2229] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:50:46.970626 8424 tablet_replica.cc:330] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: stopping tablet replica
I20260928 10:50:46.970832 8426 raft_consensus.cc:2258] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:46.970932 8425 ts_tablet_manager.cc:1939] T 9163a1a3e18a4119b45f81ced8b0af56 P a8d9ab7146de4910b532b52d842b5777: Deleting consensus metadata
I20260928 10:50:46.970893 8424 raft_consensus.cc:2229] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:50:46.971136 8424 raft_consensus.cc:2258] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:50:46.972149 8426 ts_tablet_manager.cc:1905] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:46.972245 7761 catalog_manager.cc:4841] TS a8d9ab7146de4910b532b52d842b5777 (127.5.227.131:34547): tablet 9163a1a3e18a4119b45f81ced8b0af56 (table test-table [id=8b86807113f74d08a62587704d064b94]) successfully deleted
I20260928 10:50:46.972492 8424 ts_tablet_manager.cc:1905] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:50:46.978135 8426 ts_tablet_manager.cc:1918] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:50:46.978135 8424 ts_tablet_manager.cc:1918] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:50:46.978322 8426 log.cc:1196] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/9163a1a3e18a4119b45f81ced8b0af56
I20260928 10:50:46.978336 8424 log.cc:1196] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/9163a1a3e18a4119b45f81ced8b0af56
I20260928 10:50:46.978765 8426 ts_tablet_manager.cc:1939] T 9163a1a3e18a4119b45f81ced8b0af56 P 2813f578a3fd4e48937a99d600405061: Deleting consensus metadata
I20260928 10:50:46.978767 8424 ts_tablet_manager.cc:1939] T 9163a1a3e18a4119b45f81ced8b0af56 P 290f0ca7161744a19fb563f8db1551ae: Deleting consensus metadata
I20260928 10:50:46.979861 7762 catalog_manager.cc:4841] TS 2813f578a3fd4e48937a99d600405061 (127.5.227.130:46479): tablet 9163a1a3e18a4119b45f81ced8b0af56 (table test-table [id=8b86807113f74d08a62587704d064b94]) successfully deleted
I20260928 10:50:46.979863 7764 catalog_manager.cc:4841] TS 290f0ca7161744a19fb563f8db1551ae (127.5.227.129:43401): tablet 9163a1a3e18a4119b45f81ced8b0af56 (table test-table [id=8b86807113f74d08a62587704d064b94]) successfully deleted
Sep 28 10:50:46 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592644, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:46.991817 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:46.955286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.5.227.129:43401)
0928 10:50:46.967500 (+ 12214us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:46.967510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:46.967542 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:50:46.967623 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:46.967632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:46.967755 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:46.967890 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:46.971293 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:46.972519 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:46.974595 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:46.975239 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:46.975416 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:46.985020 (+ 9604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:46.985040 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:46.985064 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:46.985110 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:46.988552 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:46.989459 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:46.989467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:46.989469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:46.989618 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:46.990061 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:46.990065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:46.990067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:46.990524 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:46.990773 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:46.991123 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 10:50:46.991338 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10682,"thread_start_us":11067,"threads_started":1}
W20260928 10:50:46.992614 7979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32976
W20260928 10:50:46.993475 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7848: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:46.993693 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 7848
I20260928 10:50:47.016424 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7988
Sep 28 10:50:47 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592644, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:50:47.032577 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:47.017339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.5.227.130:46479)
0928 10:50:47.017834 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:47.017838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:47.017852 (+ 14us) server_negotiation.cc:407] Connection header received
0928 10:50:47.017902 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:47.017906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:47.017960 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:47.018080 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:47.019284 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:47.020100 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:47.021349 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:47.021769 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:47.021906 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:47.026540 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:47.026555 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:47.026570 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:47.026610 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:47.029525 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:47.030373 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:47.030381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:47.030384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:47.030535 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:47.030955 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:47.030959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:47.030961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:47.031321 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:47.031537 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:47.032034 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 10:50:47.032197 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":243,"threads_started":1}
W20260928 10:50:47.033375 8119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43222
W20260928 10:50:47.034046 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7988: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:47.034284 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 7988
I20260928 10:50:47.062566 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8128
Sep 28 10:50:47 dist-test-slave-8w8g krb5kdc[7732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592644, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:50:47.080389 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:47.063436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.5.227.131:34547)
0928 10:50:47.064038 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:47.064042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:47.064060 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:50:47.064112 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:47.064117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:47.064183 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:47.064295 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:47.065653 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:47.066548 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:47.068068 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:47.068472 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:47.068569 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:47.073775 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:47.073794 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:47.073813 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:47.073868 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:47.077254 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:47.078147 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:47.078155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:47.078158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:47.078315 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:47.078784 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:47.078787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:47.078789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:47.079170 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:47.079414 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:47.079801 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 10:50:47.080001 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":369,"threads_started":1}
W20260928 10:50:47.081107 8259 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59942
W20260928 10:50:47.081802 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8128: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:47.081959 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 8128
I20260928 10:50:47.101854 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 7745
I20260928 10:50:47.103109 7817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:47.217450 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 7745
2026-09-28T10:50:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (12468 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:47 dist-test-slave-8w8g krb5kdc[8434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:47 dist-test-slave-8w8g krb5kdc[8434](info): set up 2 sockets
Sep 28 10:50:47 dist-test-slave-8w8g krb5kdc[8434](info): commencing operation
krb5kdc: starting...
W20260928 10:50:48.324965 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:48 dist-test-slave-8w8g krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:48.551810 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41285
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:38339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:41285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:48.899658 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:48.900104 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:48.900271 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:48.907694 8447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:48.907908 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:48.908010 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:48.908113 8447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:48.908200 8447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:48.918581 8447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:38339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:41285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:48.920115 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:48.921698 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:48.934512 8452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:48.934530 8453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:48.934708 8455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:48.935444 8447 server_base.cc:1034] running on GCE node
I20260928 10:50:48.936383 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:48.937889 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:48.939165 8447 hybrid_clock.cc:648] HybridClock initialized: now 1790592648939092 us; error 42 us; skew 500 ppm
I20260928 10:50:48.939558 8447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:48 dist-test-slave-8w8g krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:48.942991 8447 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:48.944706 8447 webserver.cc:460] Webserver started at http://127.5.227.190:39179/ using document root <none> and password file <none>
I20260928 10:50:48.945452 8447 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:48.945600 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:48.945921 8447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:48.949045 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "6c3780e1c8a547b49a677b031b489fa7"
format_stamp: "Formatted at 2026-09-28 10:50:48 on dist-test-slave-8w8g"
server_key: "f2afab3dea011d7fed126b946d30561f"
server_key_iv: "fbb64a374b5cbdb6421644a99e963c0b"
server_key_version: "encryptionkey@0"
I20260928 10:50:48.949862 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "6c3780e1c8a547b49a677b031b489fa7"
format_stamp: "Formatted at 2026-09-28 10:50:48 on dist-test-slave-8w8g"
server_key: "f2afab3dea011d7fed126b946d30561f"
server_key_iv: "fbb64a374b5cbdb6421644a99e963c0b"
server_key_version: "encryptionkey@0"
I20260928 10:50:48.955519 8447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:50:48.959439 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:48.961146 8447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:50:48.961409 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "6c3780e1c8a547b49a677b031b489fa7"
format_stamp: "Formatted at 2026-09-28 10:50:48 on dist-test-slave-8w8g"
server_key: "f2afab3dea011d7fed126b946d30561f"
server_key_iv: "fbb64a374b5cbdb6421644a99e963c0b"
server_key_version: "encryptionkey@0"
I20260928 10:50:48.961665 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:48.979306 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:48.984094 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:48.984403 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:49.003536 8447 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:41285
I20260928 10:50:49.003597 8513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:41285 every 8 connection(s)
I20260928 10:50:49.005021 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:49.009024 8514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:49.013622 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 8447
I20260928 10:50:49.013846 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:49.014693 6030 external_mini_cluster.cc:1428] Setting key d8858117c02b3755c73841be471a7c35
I20260928 10:50:49.021245 8514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7: Bootstrap starting.
I20260928 10:50:49.025354 8514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:49.026618 8514 log.cc:824] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:49.029556 8514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7: No bootstrap required, opened a new log
Sep 28 10:50:49 dist-test-slave-8w8g krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:49.040807 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:49.016989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.5.227.190:41285)
0928 10:50:49.017979 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:49.017991 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:49.018036 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:50:49.019930 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:49.019954 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:49.020366 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:49.020699 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:49.022816 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:49.024300 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:49.026162 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:49.027001 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:49.027250 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:49.031229 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:49.031246 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:49.031269 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:49.031322 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:49.035794 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:49.036592 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:49.036601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:49.036611 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:49.036836 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:49.037309 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:49.037314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:49.037318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:49.038106 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:49.038415 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:49.038869 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 10:50:49.039350 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":377,"threads_started":1}
I20260928 10:50:49.040601 8514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3780e1c8a547b49a677b031b489fa7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41285 } }
I20260928 10:50:49.041052 8514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:49.041167 8514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3780e1c8a547b49a677b031b489fa7, State: Initialized, Role: FOLLOWER
I20260928 10:50:49.041826 8514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3780e1c8a547b49a677b031b489fa7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41285 } }
I20260928 10:50:49.042105 8514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:49.042299 8514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:49.042528 8514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:49.045058 8514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3780e1c8a547b49a677b031b489fa7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41285 } }
I20260928 10:50:49.045568 8514 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c3780e1c8a547b49a677b031b489fa7; no voters:
I20260928 10:50:49.046135 8514 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:49.046295 8519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:49.047010 8519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3780e1c8a547b49a677b031b489fa7, State: Running, Role: LEADER
I20260928 10:50:49.047534 8519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3780e1c8a547b49a677b031b489fa7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41285 } }
I20260928 10:50:49.048352 8514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:49.050731 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c3780e1c8a547b49a677b031b489fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3780e1c8a547b49a677b031b489fa7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41285 } } }
I20260928 10:50:49.050963 8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3780e1c8a547b49a677b031b489fa7. Latest consensus state: current_term: 1 leader_uuid: "6c3780e1c8a547b49a677b031b489fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3780e1c8a547b49a677b031b489fa7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41285 } } }
I20260928 10:50:49.051262 8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:49.051236 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:49.053138 8524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:49.058511 8524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:49.068565 8524 catalog_manager.cc:1347] Generated new cluster ID: 625a855506ec4c95a9ed673c4b700d6c
I20260928 10:50:49.068718 8524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:49.081982 8524 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:49.083508 8524 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:49.094750 8524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3780e1c8a547b49a677b031b489fa7: Generated new TSK 0
I20260928 10:50:49.096205 8524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:50:49.150316 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41285
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:38339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:49.517519 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:49.517849 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:49.517953 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:49.526036 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:49.526198 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:49.526328 8540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:50:49.537727 8540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:38339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:49.539086 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:49.540866 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:49.556428 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:49.556670 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:49.556761 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:49.557745 8540 server_base.cc:1034] running on GCE node
I20260928 10:50:49.558527 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:49.559593 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:49.560858 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790592649560798 us; error 54 us; skew 500 ppm
I20260928 10:50:49.561388 8540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:49 dist-test-slave-8w8g krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592649, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:49.565032 8540 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:50:49.566901 8540 webserver.cc:460] Webserver started at http://127.5.227.129:42371/ using document root <none> and password file <none>
I20260928 10:50:49.567660 8540 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:49.567848 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:49.568183 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:49.572554 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "e67790a736bd4815b42f6c3661257150"
format_stamp: "Formatted at 2026-09-28 10:50:49 on dist-test-slave-8w8g"
server_key: "267b5e0d3876d5929f6d081886f5c76d"
server_key_iv: "7ebde1eff24192951bf7c4c982f24b47"
server_key_version: "encryptionkey@0"
I20260928 10:50:49.573738 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "e67790a736bd4815b42f6c3661257150"
format_stamp: "Formatted at 2026-09-28 10:50:49 on dist-test-slave-8w8g"
server_key: "267b5e0d3876d5929f6d081886f5c76d"
server_key_iv: "7ebde1eff24192951bf7c4c982f24b47"
server_key_version: "encryptionkey@0"
I20260928 10:50:49.581084 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 10:50:49.585328 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:49.587349 8540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260928 10:50:49.587599 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "e67790a736bd4815b42f6c3661257150"
format_stamp: "Formatted at 2026-09-28 10:50:49 on dist-test-slave-8w8g"
server_key: "267b5e0d3876d5929f6d081886f5c76d"
server_key_iv: "7ebde1eff24192951bf7c4c982f24b47"
server_key_version: "encryptionkey@0"
I20260928 10:50:49.587893 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:49.607388 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:49.612344 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:49.612661 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:49.613765 8540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:49.615676 8540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:49.615811 8540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 10:50:49.615967 8540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:49.616017 8540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:49.652642 8540 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:41231
I20260928 10:50:49.652655 8667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:41231 every 8 connection(s)
I20260928 10:50:49.654053 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:50:49.663589 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 8540
I20260928 10:50:49.663851 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:50:49.664705 6030 external_mini_cluster.cc:1428] Setting key 0c517427125cffb8b5472232acdfed47
Sep 28 10:50:49 dist-test-slave-8w8g krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592649, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:49.679373 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:49.657304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:49013 (local address 127.5.227.190:41285)
0928 10:50:49.657810 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:49.657818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:49.659079 (+ 1261us) server_negotiation.cc:407] Connection header received
0928 10:50:49.661191 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:49.661196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:49.661283 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:49.661385 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:49.663571 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:49.664802 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:49.667413 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:49.668145 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:49.668469 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:49.673005 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:49.673012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:49.673015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:49.673083 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:49.676009 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:49.676887 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:49.676893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:49.676899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:49.677028 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:49.677494 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:49.677501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:49.677505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:49.677792 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:49.677948 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:49.678753 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 10:50:49.679001 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260928 10:50:49.680714 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:49.657965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41285 (local address 127.5.227.129:49013)
0928 10:50:49.658886 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:49.658929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:49.660867 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:49.661703 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:49.661712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:49.662520 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:49.663227 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:49.663251 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:49.665073 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:49.665078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:49.667226 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:49.667243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:49.668567 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:49.668573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:49.668830 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:49.669881 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:49.669908 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:49.672697 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:49.676207 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:49.676219 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:49.676239 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:49.676742 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:49.677153 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:49.677158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:49.677162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:49.677374 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:49.677936 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:49.677945 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:49.678349 (+ 404us) client_negotiation.cc:769] Sending connection context
0928 10:50:49.678663 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 10:50:49.679268 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":446,"threads_started":1}
I20260928 10:50:49.682755 8668 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41285
I20260928 10:50:49.683157 8668 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:49.684267 8668 heartbeater.cc:511] Master 127.5.227.190:41285 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:50:49.687563 8479 ts_manager.cc:194] Registered new tserver with Master: e67790a736bd4815b42f6c3661257150 (127.5.227.129:41231)
I20260928 10:50:49.689150 8479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:49013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:50:49.725630 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41285
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:38339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:50.128036 8674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:50.128386 8674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:50.128485 8674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:50.138437 8674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:50.138587 8674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:50.138754 8674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:50:50.151191 8674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:38339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:50.152837 8674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:50.155134 8674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:50.172314 8680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:50.172391 8682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:50.172794 8679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:50.175068 8674 server_base.cc:1034] running on GCE node
I20260928 10:50:50.176491 8674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:50.178064 8674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:50.179486 8674 hybrid_clock.cc:648] HybridClock initialized: now 1790592650179431 us; error 44 us; skew 500 ppm
I20260928 10:50:50.180284 8674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:50 dist-test-slave-8w8g krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592650, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:50.185451 8674 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:50:50.187395 8674 webserver.cc:460] Webserver started at http://127.5.227.130:46073/ using document root <none> and password file <none>
I20260928 10:50:50.188333 8674 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:50.188504 8674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:50.188863 8674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:50.193116 8674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "f66978f110174942bed6cac0ae75db9c"
format_stamp: "Formatted at 2026-09-28 10:50:50 on dist-test-slave-8w8g"
server_key: "8b68ac3ea2b57d2e86b5843ec07bcf63"
server_key_iv: "f6d5fa7024c6bdceab62c953cc746153"
server_key_version: "encryptionkey@0"
I20260928 10:50:50.194439 8674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "f66978f110174942bed6cac0ae75db9c"
format_stamp: "Formatted at 2026-09-28 10:50:50 on dist-test-slave-8w8g"
server_key: "8b68ac3ea2b57d2e86b5843ec07bcf63"
server_key_iv: "f6d5fa7024c6bdceab62c953cc746153"
server_key_version: "encryptionkey@0"
I20260928 10:50:50.201143 8674 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:50:50.205974 8689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:50.208555 8674 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 10:50:50.208834 8674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "f66978f110174942bed6cac0ae75db9c"
format_stamp: "Formatted at 2026-09-28 10:50:50 on dist-test-slave-8w8g"
server_key: "8b68ac3ea2b57d2e86b5843ec07bcf63"
server_key_iv: "f6d5fa7024c6bdceab62c953cc746153"
server_key_version: "encryptionkey@0"
I20260928 10:50:50.209101 8674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:50.238590 8674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:50.245244 8674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:50.245719 8674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:50.247027 8674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:50.248772 8674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:50.248900 8674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:50.249070 8674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:50.249150 8674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:50.294883 8674 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:36557
I20260928 10:50:50.294927 8801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:36557 every 8 connection(s)
I20260928 10:50:50.296636 8674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:50:50.303454 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 8674
I20260928 10:50:50.303673 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:50:50.304190 6030 external_mini_cluster.cc:1428] Setting key a1428614889f5704ac9fae14ea51e549
Sep 28 10:50:50 dist-test-slave-8w8g krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592650, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:50:50.327031 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:50.300087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:35401 (local address 127.5.227.190:41285)
0928 10:50:50.300812 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:50.300817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:50.301807 (+ 990us) server_negotiation.cc:407] Connection header received
0928 10:50:50.303893 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:50.303899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:50.304000 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:50.304139 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:50.307124 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.308258 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.310245 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.311069 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.311240 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:50.317315 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:50.317326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:50.317339 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:50.317394 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:50.321675 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:50.323557 (+ 1882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:50.323566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:50.323573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:50.323864 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:50.324561 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:50.324569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:50.324574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:50.325011 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:50.325204 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:50.326079 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 10:50:50.326507 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":461,"threads_started":1}
I20260928 10:50:50.327780 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:50.300784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41285 (local address 127.5.227.130:35401)
0928 10:50:50.301576 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:50.301618 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:50.303472 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:50.304599 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:50.304616 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:50.305667 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:50.306777 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:50.306813 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.308482 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.308486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:50.310025 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:50.310044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.311399 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.311406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:50.311666 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:50.312812 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:50.312870 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:50.316949 (+ 4079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:50.322023 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:50.322039 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:50.322065 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:50.323158 (+ 1093us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:50.324118 (+ 960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:50.324123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:50.324125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:50.324395 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:50.325204 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:50.325214 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:50.325624 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 10:50:50.325897 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 10:50:50.326348 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":320,"threads_started":1}
I20260928 10:50:50.329811 8802 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41285
I20260928 10:50:50.330168 8802 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:50.330909 8802 heartbeater.cc:511] Master 127.5.227.190:41285 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:50:50.332929 8479 ts_manager.cc:194] Registered new tserver with Master: f66978f110174942bed6cac0ae75db9c (127.5.227.130:36557)
I20260928 10:50:50.334694 8479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:35401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:50:50.365855 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41285
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:38339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:50:50.693212 8668 heartbeater.cc:503] Master 127.5.227.190:41285 was elected leader, sending a full tablet report...
W20260928 10:50:50.777984 8808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:50.778483 8808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:50.778702 8808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:50.789252 8808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:50.789405 8808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:50.789595 8808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:50:50.801105 8808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:38339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:50.802712 8808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:50.804317 8808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:50.820436 8816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:50.820463 8813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:50.820484 8814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:50.821660 8808 server_base.cc:1034] running on GCE node
I20260928 10:50:50.822368 8808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:50.823432 8808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:50.824903 8808 hybrid_clock.cc:648] HybridClock initialized: now 1790592650824838 us; error 51 us; skew 500 ppm
I20260928 10:50:50.825333 8808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:50 dist-test-slave-8w8g krb5kdc[8434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592650, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:50.829943 8808 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:50:50.831904 8808 webserver.cc:460] Webserver started at http://127.5.227.131:42317/ using document root <none> and password file <none>
I20260928 10:50:50.832723 8808 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:50.832894 8808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:50.833400 8808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:50.837000 8808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "3cf808d299eb48b7a68702f9515e281f"
format_stamp: "Formatted at 2026-09-28 10:50:50 on dist-test-slave-8w8g"
server_key: "2e8a7f631c727e1c1099e25a4dbac961"
server_key_iv: "84f927b0d72426a5237ba10ea6b3ca29"
server_key_version: "encryptionkey@0"
I20260928 10:50:50.837926 8808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "3cf808d299eb48b7a68702f9515e281f"
format_stamp: "Formatted at 2026-09-28 10:50:50 on dist-test-slave-8w8g"
server_key: "2e8a7f631c727e1c1099e25a4dbac961"
server_key_iv: "84f927b0d72426a5237ba10ea6b3ca29"
server_key_version: "encryptionkey@0"
I20260928 10:50:50.844341 8808 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:50:50.848532 8823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:50.851101 8808 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:50:50.851504 8808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "3cf808d299eb48b7a68702f9515e281f"
format_stamp: "Formatted at 2026-09-28 10:50:50 on dist-test-slave-8w8g"
server_key: "2e8a7f631c727e1c1099e25a4dbac961"
server_key_iv: "84f927b0d72426a5237ba10ea6b3ca29"
server_key_version: "encryptionkey@0"
I20260928 10:50:50.851823 8808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:50.873648 8808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:50.880268 8808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:50.880693 8808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:50.881937 8808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:50.883466 8808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:50.883584 8808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:50.883773 8808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:50.883852 8808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 10:50:50.929958 8808 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:36007
I20260928 10:50:50.929971 8935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:36007 every 8 connection(s)
I20260928 10:50:50.931557 8808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:50:50.932469 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 8808
I20260928 10:50:50.932703 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:50:50.933349 6030 external_mini_cluster.cc:1428] Setting key 04a05549365854363ab3c8706790e34b
Sep 28 10:50:50 dist-test-slave-8w8g krb5kdc[8434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592650, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:50.961354 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:50.935956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:46697 (local address 127.5.227.190:41285)
0928 10:50:50.937116 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:50.937121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:50.939575 (+ 2454us) server_negotiation.cc:407] Connection header received
0928 10:50:50.942266 (+ 2691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:50.942272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:50.942543 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:50.942970 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:50.945440 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.946252 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.948173 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.948629 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.948792 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:50.954389 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:50.954394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:50.954398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:50.954435 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:50.957629 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:50.958531 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:50.958536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:50.958538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:50.958665 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:50.959122 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:50.959126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:50.959129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:50.959409 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:50.959579 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:50.960573 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 10:50:50.960831 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":844,"threads_started":1}
I20260928 10:50:50.962659 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:50.936845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41285 (local address 127.5.227.131:46697)
0928 10:50:50.938536 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:50.938600 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:50.941081 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:50.943480 (+ 2399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:50.943492 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:50.944312 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:50.945177 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:50.945213 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.946449 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.946453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:50.947926 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:50.947941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:50.948931 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:50.948936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:50.949335 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:50.950350 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:50.950383 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:50.954123 (+ 3740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:50.957819 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:50.957830 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:50.957859 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:50.958379 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:50.958783 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:50.958788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:50.958792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:50.958989 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:50.959546 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:50.959556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:50.960015 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 10:50:50.960458 (+ 443us) client_negotiation.cc:240] Negotiation successful
0928 10:50:50.961153 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":576,"threads_started":1}
I20260928 10:50:50.964620 8936 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41285
I20260928 10:50:50.964998 8936 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:50.965862 8936 heartbeater.cc:511] Master 127.5.227.190:41285 requested a full tablet report, sending...
I20260928 10:50:50.967528 8479 ts_manager.cc:194] Registered new tserver with Master: 3cf808d299eb48b7a68702f9515e281f (127.5.227.131:36007)
I20260928 10:50:50.968422 8479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:46697
I20260928 10:50:50.979934 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:50:51.016227 8938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:50:51.013185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.5.227.190:41285)
0928 10:50:51.013419 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:51.013429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:51.013958 (+ 529us) server_negotiation.cc:407] Connection header received
0928 10:50:51.014094 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:51.014098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:51.014218 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:51.014411 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:51.016023 (+ 1612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33572: BlockingRecv error: recv got EOF from 127.0.0.1:33572 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:50:51.020149 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8540
W20260928 10:50:51.024051 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8540: Not authorized: Client connection negotiation failed: client connection to 127.5.227.129:41231: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:50:51.024477 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 8540
I20260928 10:50:51.042565 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8674
W20260928 10:50:51.046005 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8674: Not authorized: Client connection negotiation failed: client connection to 127.5.227.130:36557: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:50:51.046152 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 8674
I20260928 10:50:51.064194 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8808
W20260928 10:50:51.068488 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8808: Not authorized: Client connection negotiation failed: client connection to 127.5.227.131:36007: server requires authentication, but client does not have Kerberos credentials available
W20260928 10:50:51.068723 8950 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:50:51.065472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.5.227.131:36007)
0928 10:50:51.066389 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:51.066397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:51.066430 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:50:51.066492 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:51.066498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:51.066780 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:51.066999 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:51.068410 (+ 1411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56528: BlockingRecv error: recv got EOF from 127.0.0.1:56528 (error 108)
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":483,"threads_started":1}
I20260928 10:50:51.068810 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 8808
I20260928 10:50:51.085098 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8447
I20260928 10:50:51.086859 8509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:51.200263 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 8447
2026-09-28T10:50:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3981 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:51 dist-test-slave-8w8g krb5kdc[8954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:51 dist-test-slave-8w8g krb5kdc[8954](info): set up 2 sockets
Sep 28 10:50:51 dist-test-slave-8w8g krb5kdc[8954](info): commencing operation
krb5kdc: starting...
W20260928 10:50:52.406040 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:52 dist-test-slave-8w8g krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:52.656854 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:35901
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:41401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:35901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:53.049990 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:53.050392 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:53.050583 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:53.058676 8967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:53.058902 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:53.059029 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:53.059142 8967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:53.059244 8967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:53.070029 8967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:35901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:35901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:53.071511 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:53.073084 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:53.086198 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:53.086287 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:53.086316 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:53.087129 8967 server_base.cc:1034] running on GCE node
I20260928 10:50:53.088083 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:53.090049 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:53.091296 8967 hybrid_clock.cc:648] HybridClock initialized: now 1790592653091190 us; error 103 us; skew 500 ppm
I20260928 10:50:53.091666 8967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:53 dist-test-slave-8w8g krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:53.101361 8967 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:53.103143 8967 webserver.cc:460] Webserver started at http://127.5.227.190:39013/ using document root <none> and password file <none>
I20260928 10:50:53.103977 8967 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:53.104115 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:53.104453 8967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:53.107722 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "5f1af0dad4f2444cbe734afe68ec4f62"
format_stamp: "Formatted at 2026-09-28 10:50:53 on dist-test-slave-8w8g"
server_key: "e7886f08d8169d4075a36aefa04ba9d8"
server_key_iv: "5a0db101294a7d5cc37ade64e33d2dc4"
server_key_version: "encryptionkey@0"
I20260928 10:50:53.108760 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "5f1af0dad4f2444cbe734afe68ec4f62"
format_stamp: "Formatted at 2026-09-28 10:50:53 on dist-test-slave-8w8g"
server_key: "e7886f08d8169d4075a36aefa04ba9d8"
server_key_iv: "5a0db101294a7d5cc37ade64e33d2dc4"
server_key_version: "encryptionkey@0"
I20260928 10:50:53.114688 8967 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 10:50:53.119083 8982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:53.121025 8967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:50:53.121258 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "5f1af0dad4f2444cbe734afe68ec4f62"
format_stamp: "Formatted at 2026-09-28 10:50:53 on dist-test-slave-8w8g"
server_key: "e7886f08d8169d4075a36aefa04ba9d8"
server_key_iv: "5a0db101294a7d5cc37ade64e33d2dc4"
server_key_version: "encryptionkey@0"
I20260928 10:50:53.121495 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:53.154346 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:53.159978 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:53.160483 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:53.186520 8967 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:35901
I20260928 10:50:53.186555 9033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:35901 every 8 connection(s)
I20260928 10:50:53.188277 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:53.191649 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 8967
I20260928 10:50:53.191946 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:53.192932 6030 external_mini_cluster.cc:1428] Setting key cda24522f23cb76a5f8940c58a6183f2
I20260928 10:50:53.193492 9034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:50:53 dist-test-slave-8w8g krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:53.211680 9034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62: Bootstrap starting.
I20260928 10:50:53.216307 9034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:53.217841 9034 log.cc:824] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:53.222517 9034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62: No bootstrap required, opened a new log
I20260928 10:50:53.224889 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:53.195157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.5.227.190:35901)
0928 10:50:53.196223 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:53.196242 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:53.196301 (+ 59us) server_negotiation.cc:407] Connection header received
0928 10:50:53.198021 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:53.198054 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:53.198740 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:53.199307 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:53.201408 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:53.204044 (+ 2636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:53.205795 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:53.207664 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:53.207951 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:53.212960 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:53.212973 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:53.212996 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:53.213054 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:53.217755 (+ 4701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:53.219708 (+ 1953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:53.219717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:53.219724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:53.220287 (+ 563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:53.220864 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:53.220869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:53.220872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:53.221643 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:53.221954 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:53.222553 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 10:50:53.223115 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":465,"threads_started":1}
I20260928 10:50:53.231885 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:50:53.234310 9034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 35901 } }
I20260928 10:50:53.234848 9034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:53.235020 9034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1af0dad4f2444cbe734afe68ec4f62, State: Initialized, Role: FOLLOWER
I20260928 10:50:53.235970 9034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 35901 } }
I20260928 10:50:53.236271 9034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:53.236421 9034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:53.236608 9034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:53.239492 9034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 35901 } }
I20260928 10:50:53.240337 9034 leader_election.cc:302] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f1af0dad4f2444cbe734afe68ec4f62; no voters:
I20260928 10:50:53.241230 9034 leader_election.cc:288] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:53.241428 9040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:53.242712 9040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [term 1 LEADER]: Becoming Leader. State: Replica: 5f1af0dad4f2444cbe734afe68ec4f62, State: Running, Role: LEADER
I20260928 10:50:53.243431 9040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 35901 } }
I20260928 10:50:53.243857 9034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:53.247035 9042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f1af0dad4f2444cbe734afe68ec4f62. Latest consensus state: current_term: 1 leader_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 35901 } } }
I20260928 10:50:53.247026 9041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1af0dad4f2444cbe734afe68ec4f62" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 35901 } } }
I20260928 10:50:53.247829 9041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:53.247829 9042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:53.260557 9046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:53.267620 9046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:53.274412 9058 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:50:53.281200 9046 catalog_manager.cc:1347] Generated new cluster ID: 83359d6de13b4f659f976daf979dcb8a
I20260928 10:50:53.281330 9046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:53.290297 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:53.281723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.5.227.190:35901)
0928 10:50:53.282578 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:53.282586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:53.282607 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:50:53.282830 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:53.282836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:53.282912 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:53.283040 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:53.284117 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:53.285062 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:53.286643 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:53.287130 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:53.287295 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:53.288556 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:53.288568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:53.288572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:50:53.289330 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:53.289385 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:50:53.289412 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:53.289550 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:53.289703 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 10:50:53.289820 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731}
I20260928 10:50:53.322162 9046 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:53.324905 9046 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:53.345506 9046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f1af0dad4f2444cbe734afe68ec4f62: Generated new TSK 0
I20260928 10:50:53.346707 9046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:50:53.359812 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 8967
I20260928 10:50:53.361589 9029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:53.462311 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 8967
2026-09-28T10:50:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2256 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:53 dist-test-slave-8w8g krb5kdc[9068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:53 dist-test-slave-8w8g krb5kdc[9068](info): set up 2 sockets
Sep 28 10:50:53 dist-test-slave-8w8g krb5kdc[9068](info): commencing operation
krb5kdc: starting...
W20260928 10:50:54.636479 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:54 dist-test-slave-8w8g krb5kdc[9068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:54.852392 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:32973
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:32973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:55.203711 9081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:55.204098 9081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:55.204288 9081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:55.212028 9081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:55.212149 9081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:55.212196 9081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:55.212281 9081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:55.212359 9081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:55.222756 9081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:32973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:32973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:55.224289 9081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:55.225630 9081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:55.239234 9086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:55.239506 9087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:55.240054 9089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:55.240443 9081 server_base.cc:1034] running on GCE node
I20260928 10:50:55.241604 9081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:55.243407 9081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:55.244721 9081 hybrid_clock.cc:648] HybridClock initialized: now 1790592655244686 us; error 104 us; skew 500 ppm
I20260928 10:50:55.245113 9081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:55 dist-test-slave-8w8g krb5kdc[9068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:55.248667 9081 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:55.250538 9081 webserver.cc:460] Webserver started at http://127.5.227.190:40127/ using document root <none> and password file <none>
I20260928 10:50:55.251334 9081 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:55.251493 9081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:55.251885 9081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:55.254976 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "1d3b32d31f5c4941a3f483b5882c84f8"
format_stamp: "Formatted at 2026-09-28 10:50:55 on dist-test-slave-8w8g"
server_key: "a807e00892a439c6550c93f51a7c0f97"
server_key_iv: "aeb7c88eff110f9a0a36900fd3029cf4"
server_key_version: "encryptionkey@0"
I20260928 10:50:55.255934 9081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "1d3b32d31f5c4941a3f483b5882c84f8"
format_stamp: "Formatted at 2026-09-28 10:50:55 on dist-test-slave-8w8g"
server_key: "a807e00892a439c6550c93f51a7c0f97"
server_key_iv: "aeb7c88eff110f9a0a36900fd3029cf4"
server_key_version: "encryptionkey@0"
I20260928 10:50:55.261996 9081 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:50:55.266031 9096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:55.267697 9081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:50:55.267946 9081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "1d3b32d31f5c4941a3f483b5882c84f8"
format_stamp: "Formatted at 2026-09-28 10:50:55 on dist-test-slave-8w8g"
server_key: "a807e00892a439c6550c93f51a7c0f97"
server_key_iv: "aeb7c88eff110f9a0a36900fd3029cf4"
server_key_version: "encryptionkey@0"
I20260928 10:50:55.268177 9081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:55.287439 9081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:55.293188 9081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:55.293538 9081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:55.315019 9081 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:32973
I20260928 10:50:55.315203 9147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:32973 every 8 connection(s)
I20260928 10:50:55.316581 9081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:55.320923 9148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:55.324622 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9081
I20260928 10:50:55.324869 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:55.325841 6030 external_mini_cluster.cc:1428] Setting key 822dca22b88e13ec7f26b9df305625bd
I20260928 10:50:55.335917 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8: Bootstrap starting.
I20260928 10:50:55.340077 9148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:55.341759 9148 log.cc:824] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:50:55 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:55.345526 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8: No bootstrap required, opened a new log
I20260928 10:50:55.353430 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:55.328447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.5.227.190:32973)
0928 10:50:55.329304 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:55.329317 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:55.329360 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:50:55.331031 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:55.331058 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:55.331524 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:55.331981 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:55.333825 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:55.335325 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:55.337206 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:55.338004 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:55.338169 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:55.343925 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:55.343939 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:55.343967 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:55.344006 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:55.347980 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:55.348884 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:55.348897 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:55.348912 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:55.349167 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:55.349746 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:55.349755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:55.349758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:55.350560 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:55.350906 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:55.351301 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 10:50:55.351854 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":412,"threads_started":1}
I20260928 10:50:55.356325 9148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32973 } }
I20260928 10:50:55.356801 9148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:55.356938 9148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3b32d31f5c4941a3f483b5882c84f8, State: Initialized, Role: FOLLOWER
I20260928 10:50:55.357659 9148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32973 } }
I20260928 10:50:55.357939 9148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:55.358104 9148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:55.358459 9148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:55.360924 9148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32973 } }
I20260928 10:50:55.361467 9148 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d3b32d31f5c4941a3f483b5882c84f8; no voters:
I20260928 10:50:55.362118 9148 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:55.362403 9153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:55.363582 9153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3b32d31f5c4941a3f483b5882c84f8, State: Running, Role: LEADER
I20260928 10:50:55.364364 9153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32973 } }
I20260928 10:50:55.364673 9148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:55.367952 9154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32973 } } }
I20260928 10:50:55.368371 9154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:55.367493 9155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3b32d31f5c4941a3f483b5882c84f8. Latest consensus state: current_term: 1 leader_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3b32d31f5c4941a3f483b5882c84f8" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32973 } } }
I20260928 10:50:55.368937 9155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:55.370857 9159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:55.379885 9159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:55.390470 9159 catalog_manager.cc:1347] Generated new cluster ID: 5a1f3a74f63542d5a6f4b54473b9b8d8
I20260928 10:50:55.390673 9159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:55.412699 9159 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:55.414721 9159 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:55.427596 9159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3b32d31f5c4941a3f483b5882c84f8: Generated new TSK 0
I20260928 10:50:55.428776 9159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:50:55.490449 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32973
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:55.870998 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:55.871372 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:55.871592 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:55.879927 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:55.880098 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:55.880337 9174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:50:55.890592 9174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:55.892014 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:55.893865 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:55.912128 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:55.912170 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:55.917387 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:55.919005 9174 server_base.cc:1034] running on GCE node
I20260928 10:50:55.919720 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:55.920892 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:55.922134 9174 hybrid_clock.cc:648] HybridClock initialized: now 1790592655922085 us; error 39 us; skew 500 ppm
I20260928 10:50:55.922513 9174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:55 dist-test-slave-8w8g krb5kdc[9068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:55.926640 9174 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:50:55.928535 9174 webserver.cc:460] Webserver started at http://127.5.227.129:36429/ using document root <none> and password file <none>
I20260928 10:50:55.929317 9174 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:55.929454 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:55.929800 9174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:55.933154 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "498dd7aa48034245ac795b1e37aeeba2"
format_stamp: "Formatted at 2026-09-28 10:50:55 on dist-test-slave-8w8g"
server_key: "125db2a4df40055048f9bc2e5a9446c4"
server_key_iv: "df576076f808c7e6497dc2c2f70f2d1a"
server_key_version: "encryptionkey@0"
I20260928 10:50:55.934016 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "498dd7aa48034245ac795b1e37aeeba2"
format_stamp: "Formatted at 2026-09-28 10:50:55 on dist-test-slave-8w8g"
server_key: "125db2a4df40055048f9bc2e5a9446c4"
server_key_iv: "df576076f808c7e6497dc2c2f70f2d1a"
server_key_version: "encryptionkey@0"
I20260928 10:50:55.941010 9174 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260928 10:50:55.945232 9189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:55.946991 9174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 10:50:55.947250 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "498dd7aa48034245ac795b1e37aeeba2"
format_stamp: "Formatted at 2026-09-28 10:50:55 on dist-test-slave-8w8g"
server_key: "125db2a4df40055048f9bc2e5a9446c4"
server_key_iv: "df576076f808c7e6497dc2c2f70f2d1a"
server_key_version: "encryptionkey@0"
I20260928 10:50:55.947491 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:55.974956 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:55.979828 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:55.980171 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:55.981238 9174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:55.982736 9174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:55.982852 9174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:55.983002 9174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:55.983073 9174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:56.021523 9174 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:38447
I20260928 10:50:56.021627 9301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:38447 every 8 connection(s)
I20260928 10:50:56.022926 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:50:56.029799 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9174
I20260928 10:50:56.030055 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:50:56.030872 6030 external_mini_cluster.cc:1428] Setting key 3877988ef56a2f7a62d3960470be6cee
Sep 28 10:50:56 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:56.047860 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:56.025947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:52469 (local address 127.5.227.190:32973)
0928 10:50:56.026427 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:56.026432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:56.027578 (+ 1146us) server_negotiation.cc:407] Connection header received
0928 10:50:56.029974 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:56.029980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:56.030095 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:56.030236 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:56.032352 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.033352 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.035715 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.036486 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.036659 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:56.041261 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:56.041267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:56.041271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:56.041305 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:56.044088 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:56.045132 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:56.045138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:56.045141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:56.045264 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:56.045801 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:56.045804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:56.045806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:56.046072 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:56.046197 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:56.047156 (+ 959us) server_negotiation.cc:299] Negotiation successful
0928 10:50:56.047414 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
I20260928 10:50:56.049450 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:56.026575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:32973 (local address 127.5.227.129:52469)
0928 10:50:56.027393 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:56.027436 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:56.029532 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:56.030515 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:56.030525 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:56.031177 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:56.031999 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:56.032046 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.033638 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.033644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:56.035496 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:56.035514 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.036781 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.036787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:56.037117 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:56.037951 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:56.037981 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:56.040962 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:56.044283 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:56.044296 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:56.044318 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:56.044981 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:56.045392 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:56.045400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:56.045407 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:56.045667 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:56.046230 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:56.046241 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:56.046738 (+ 497us) client_negotiation.cc:769] Sending connection context
0928 10:50:56.047054 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 10:50:56.047723 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":324,"threads_started":1}
I20260928 10:50:56.051200 9302 heartbeater.cc:347] Connected to a master server at 127.5.227.190:32973
I20260928 10:50:56.051589 9302 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:50:56.052489 9302 heartbeater.cc:511] Master 127.5.227.190:32973 requested a full tablet report, sending...
I20260928 10:50:56.055138 9113 ts_manager.cc:194] Registered new tserver with Master: 498dd7aa48034245ac795b1e37aeeba2 (127.5.227.129:38447)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:50:56.057757 9113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:52469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:50:56.084867 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32973
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:56.446158 9308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:56.446462 9308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:56.446556 9308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:56.454473 9308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:56.454617 9308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:56.454737 9308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:50:56.465305 9308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:56.466696 9308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:56.468700 9308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:56.485196 9316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:56.485230 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:56.485260 9314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:56.487021 9308 server_base.cc:1034] running on GCE node
I20260928 10:50:56.487833 9308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:56.488987 9308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:56.490267 9308 hybrid_clock.cc:648] HybridClock initialized: now 1790592656490206 us; error 63 us; skew 500 ppm
I20260928 10:50:56.490720 9308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:56 dist-test-slave-8w8g krb5kdc[9068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:56.494462 9308 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:50:56.496387 9308 webserver.cc:460] Webserver started at http://127.5.227.130:36269/ using document root <none> and password file <none>
I20260928 10:50:56.497177 9308 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:56.497339 9308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:56.497666 9308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:56.500912 9308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "6e1c923373744707b648bc08aaa488b7"
format_stamp: "Formatted at 2026-09-28 10:50:56 on dist-test-slave-8w8g"
server_key: "a194f4d210b2aec40a173843f54710cc"
server_key_iv: "82efc717a37547bfd94cd99fd4117326"
server_key_version: "encryptionkey@0"
I20260928 10:50:56.501801 9308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "6e1c923373744707b648bc08aaa488b7"
format_stamp: "Formatted at 2026-09-28 10:50:56 on dist-test-slave-8w8g"
server_key: "a194f4d210b2aec40a173843f54710cc"
server_key_iv: "82efc717a37547bfd94cd99fd4117326"
server_key_version: "encryptionkey@0"
I20260928 10:50:56.507781 9308 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:50:56.512395 9323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:56.513931 9308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:50:56.514171 9308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "6e1c923373744707b648bc08aaa488b7"
format_stamp: "Formatted at 2026-09-28 10:50:56 on dist-test-slave-8w8g"
server_key: "a194f4d210b2aec40a173843f54710cc"
server_key_iv: "82efc717a37547bfd94cd99fd4117326"
server_key_version: "encryptionkey@0"
I20260928 10:50:56.514405 9308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:56.535516 9308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:56.540555 9308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:56.540877 9308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:56.541934 9308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:56.543466 9308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:56.543586 9308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:56.543772 9308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:56.543856 9308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 10:50:56.583396 9308 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:33611
I20260928 10:50:56.583451 9435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:33611 every 8 connection(s)
I20260928 10:50:56.584781 9308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:50:56.588187 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9308
I20260928 10:50:56.588402 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:50:56.588872 6030 external_mini_cluster.cc:1428] Setting key 8bbedef83a9884ee203d1269df6d3ae6
Sep 28 10:50:56 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:50:56.609985 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:56.587563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:43483 (local address 127.5.227.190:32973)
0928 10:50:56.588105 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:56.588112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:56.589266 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 10:50:56.591308 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:56.591314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:56.591403 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:56.591526 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:56.593344 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.594742 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.597054 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.597692 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.597828 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:56.603238 (+ 5410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:56.603249 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:56.603252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:56.603292 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:56.606239 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:56.607293 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:56.607299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:56.607301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:56.607439 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:56.608029 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:56.608034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:56.608036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:56.608318 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:56.608488 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:56.609312 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 10:50:56.609548 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":265,"threads_started":1}
I20260928 10:50:56.611173 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:56.588185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:32973 (local address 127.5.227.130:43483)
0928 10:50:56.589073 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:56.589128 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:56.591039 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:56.591899 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:56.591908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:56.592477 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:56.593142 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:56.593162 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.594941 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.594946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:56.596861 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:56.596880 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:56.597954 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:56.597964 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:56.598340 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:56.599236 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:56.599282 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:56.602799 (+ 3517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:56.606446 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:56.606459 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:56.606604 (+ 145us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:56.607102 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:56.607574 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:56.607579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:56.607583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:56.607884 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:56.608471 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:56.608481 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:56.608933 (+ 452us) client_negotiation.cc:769] Sending connection context
0928 10:50:56.609221 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 10:50:56.609817 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":408,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:50:56.612993 9436 heartbeater.cc:347] Connected to a master server at 127.5.227.190:32973
I20260928 10:50:56.613469 9436 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:56.614372 9436 heartbeater.cc:511] Master 127.5.227.190:32973 requested a full tablet report, sending...
I20260928 10:50:56.616429 9113 ts_manager.cc:194] Registered new tserver with Master: 6e1c923373744707b648bc08aaa488b7 (127.5.227.130:33611)
I20260928 10:50:56.617605 9113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:43483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:50:56.640177 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32973
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:57.004516 9442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:57.004848 9442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:57.005007 9442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:57.013432 9442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:57.013564 9442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:57.013758 9442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:50:57.025127 9442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:57.026928 9442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:57.028607 9442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:57.044441 9447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:57.044873 9448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:57.045092 9450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:57.045439 9442 server_base.cc:1034] running on GCE node
I20260928 10:50:57.046249 9442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:57.047336 9442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:57.048655 9442 hybrid_clock.cc:648] HybridClock initialized: now 1790592657048587 us; error 62 us; skew 500 ppm
I20260928 10:50:57.049103 9442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:57.053058 9442 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:50:57.054875 9442 webserver.cc:460] Webserver started at http://127.5.227.131:40241/ using document root <none> and password file <none>
I20260928 10:50:57.056025 9442 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:57.056212 9442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:57.056591 9442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:57.059996 9442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "bc1e71192bc341a9816d3db4e1ef608a"
format_stamp: "Formatted at 2026-09-28 10:50:57 on dist-test-slave-8w8g"
server_key: "2a2edef28397f96b2a15c7a66ad32055"
server_key_iv: "a5a7c26faf2fa8e343ee4ef02126de7d"
server_key_version: "encryptionkey@0"
I20260928 10:50:57.060912 9442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "bc1e71192bc341a9816d3db4e1ef608a"
format_stamp: "Formatted at 2026-09-28 10:50:57 on dist-test-slave-8w8g"
server_key: "2a2edef28397f96b2a15c7a66ad32055"
server_key_iv: "a5a7c26faf2fa8e343ee4ef02126de7d"
server_key_version: "encryptionkey@0"
I20260928 10:50:57.061939 9302 heartbeater.cc:503] Master 127.5.227.190:32973 was elected leader, sending a full tablet report...
I20260928 10:50:57.068660 9442 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 10:50:57.074083 9457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:57.076148 9442 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260928 10:50:57.076333 9442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "bc1e71192bc341a9816d3db4e1ef608a"
format_stamp: "Formatted at 2026-09-28 10:50:57 on dist-test-slave-8w8g"
server_key: "2a2edef28397f96b2a15c7a66ad32055"
server_key_iv: "a5a7c26faf2fa8e343ee4ef02126de7d"
server_key_version: "encryptionkey@0"
I20260928 10:50:57.076591 9442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:57.102232 9442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:57.108245 9442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:57.108726 9442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:57.109877 9442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:50:57.111552 9442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:50:57.111701 9442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:57.111864 9442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:50:57.111940 9442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:57.153746 9442 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:42543
I20260928 10:50:57.153787 9569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:42543 every 8 connection(s)
I20260928 10:50:57.155164 9442 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:50:57.164356 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9442
I20260928 10:50:57.164624 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:50:57.165108 6030 external_mini_cluster.cc:1428] Setting key 0004f4d8a9bdd341003fed8c40f90a7f
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:57.183143 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:57.158779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:49175 (local address 127.5.227.190:32973)
0928 10:50:57.159304 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:57.159309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:57.161038 (+ 1729us) server_negotiation.cc:407] Connection header received
0928 10:50:57.164003 (+ 2965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:57.164010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:57.164266 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:57.164427 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:57.167003 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.168210 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.170666 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.171432 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.171632 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:57.176545 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:57.176552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:57.176555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:57.176601 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:57.179470 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.180433 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.180440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.180444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.180590 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.181225 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.181231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.181233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.181516 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:57.181722 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:57.182489 (+ 767us) server_negotiation.cc:299] Negotiation successful
0928 10:50:57.182741 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":339,"threads_started":1}
I20260928 10:50:57.184592 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:57.159662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:32973 (local address 127.5.227.131:49175)
0928 10:50:57.160807 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0928 10:50:57.160855 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 10:50:57.163618 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:50:57.164766 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:50:57.164781 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:50:57.165517 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:50:57.166790 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:57.166818 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.168386 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.168391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:57.170194 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:50:57.170225 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.171799 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.171810 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:50:57.172186 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:57.172993 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:50:57.173020 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:50:57.176243 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:50:57.179675 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:57.179687 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:57.179709 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:57.180296 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:57.180741 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:50:57.180749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:50:57.180754 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:50:57.181087 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:50:57.181661 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:50:57.181675 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:50:57.182101 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 10:50:57.182391 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 10:50:57.182979 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":523,"threads_started":1}
I20260928 10:50:57.186287 9570 heartbeater.cc:347] Connected to a master server at 127.5.227.190:32973
I20260928 10:50:57.186625 9570 heartbeater.cc:464] Registering TS with master...
I20260928 10:50:57.187407 9570 heartbeater.cc:511] Master 127.5.227.190:32973 requested a full tablet report, sending...
I20260928 10:50:57.189577 9113 ts_manager.cc:194] Registered new tserver with Master: bc1e71192bc341a9816d3db4e1ef608a (127.5.227.131:42543)
I20260928 10:50:57.190724 9113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:49175
I20260928 10:50:57.194514 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592657, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592657, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:57.252483 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:57.233357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.5.227.190:32973)
0928 10:50:57.233717 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:57.233726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:57.234096 (+ 370us) server_negotiation.cc:407] Connection header received
0928 10:50:57.234361 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:57.234367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:57.234453 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:57.234586 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:57.236953 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.238019 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.239660 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.240312 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.240539 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:57.245255 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:57.245261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:57.245265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:57.245308 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:57.248884 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.249844 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.249851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.249853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.250013 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.250501 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.250509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.250514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.251005 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:57.251300 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:57.251716 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 10:50:57.252034 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 10:50:57.253158 9113 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41642
I20260928 10:50:57.255296 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9174
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592657, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:50:57.273927 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:57.256651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.5.227.129:38447)
0928 10:50:57.257311 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:57.257318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:57.257344 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:50:57.257406 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:57.257413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:57.257671 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:57.257826 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:57.259005 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.260229 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.261474 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.262173 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.262290 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:57.267178 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:57.267194 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:57.267211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:57.267259 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:57.270600 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.271476 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.271484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.271486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.271638 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.272116 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.272123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.272127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.272547 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:57.272891 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:57.273227 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 10:50:57.273535 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":331,"threads_started":1}
W20260928 10:50:57.275429 9297 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49284
W20260928 10:50:57.276769 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9174: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:57.276985 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9174
I20260928 10:50:57.294076 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9308
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592657, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:50:57.313870 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:57.295279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.5.227.130:33611)
0928 10:50:57.295929 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:57.295937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:57.295964 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:50:57.296041 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:57.296048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:57.296375 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:57.296637 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:57.297896 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.298925 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.300431 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.301009 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.301166 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:57.305871 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:57.305885 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:57.305901 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:57.305946 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:57.309951 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.310855 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.310867 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.310869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.311019 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.311558 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.311564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.311567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.312146 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:57.312467 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:57.313238 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 10:50:57.313553 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":321,"threads_started":1}
W20260928 10:50:57.315467 9431 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45784
W20260928 10:50:57.316881 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9308: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:57.317127 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9308
I20260928 10:50:57.337354 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9442
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592657, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:50:57.356812 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:57.338469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.5.227.131:42543)
0928 10:50:57.339292 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:57.339299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:57.339328 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:50:57.339388 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:57.339395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:57.339631 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:57.339849 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:57.341539 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.342725 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.344505 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:57.345327 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:57.345469 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:57.350448 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:57.350464 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:57.350480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:57.350581 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:57.353633 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.354502 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.354510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.354513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.354683 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:57.355038 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:57.355041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:57.355043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:57.355427 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:57.355703 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:57.356124 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 10:50:57.356336 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":405,"threads_started":1}
W20260928 10:50:57.358202 9565 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41160
W20260928 10:50:57.359519 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9442: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:50:57.359690 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9442
I20260928 10:50:57.376760 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9081
I20260928 10:50:57.378053 9143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:57.482447 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9081
2026-09-28T10:50:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4023 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9588](info): set up 2 sockets
Sep 28 10:50:57 dist-test-slave-8w8g krb5kdc[9588](info): commencing operation
krb5kdc: starting...
W20260928 10:50:58.640863 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:50:58 dist-test-slave-8w8g krb5kdc[9588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:50:58.877463 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39323
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:59.270342 9601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:59.270669 9601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:59.270919 9601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:50:59.279332 9601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:50:59.279482 9601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:50:59.279532 9601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:50:59.279583 9601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:50:59.279680 9601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:50:59.292303 9601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:50:59.294001 9601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:50:59.295686 9601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:50:59.311129 9607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:59.311226 9606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:50:59.311273 9609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:50:59.312909 9601 server_base.cc:1034] running on GCE node
I20260928 10:50:59.314219 9601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:50:59.316371 9601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:50:59.317756 9601 hybrid_clock.cc:648] HybridClock initialized: now 1790592659317633 us; error 77 us; skew 500 ppm
I20260928 10:50:59.318317 9601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:50:59 dist-test-slave-8w8g krb5kdc[9588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:50:59.328840 9601 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:50:59.330722 9601 webserver.cc:460] Webserver started at http://127.5.227.190:36119/ using document root <none> and password file <none>
I20260928 10:50:59.331555 9601 fs_manager.cc:359] Metadata directory not provided
I20260928 10:50:59.331724 9601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:50:59.332137 9601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:50:59.335457 9601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "8b8e9f1ccadc455b8d23d0c739187163"
format_stamp: "Formatted at 2026-09-28 10:50:59 on dist-test-slave-8w8g"
server_key: "8c062f7d78513658430a3f941644eb85"
server_key_iv: "eb6b1237e3d2b7fb43d440d5cc4b94d9"
server_key_version: "encryptionkey@0"
I20260928 10:50:59.336470 9601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "8b8e9f1ccadc455b8d23d0c739187163"
format_stamp: "Formatted at 2026-09-28 10:50:59 on dist-test-slave-8w8g"
server_key: "8c062f7d78513658430a3f941644eb85"
server_key_iv: "eb6b1237e3d2b7fb43d440d5cc4b94d9"
server_key_version: "encryptionkey@0"
I20260928 10:50:59.342723 9601 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:50:59.347051 9616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:50:59.349242 9601 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 10:50:59.349502 9601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "8b8e9f1ccadc455b8d23d0c739187163"
format_stamp: "Formatted at 2026-09-28 10:50:59 on dist-test-slave-8w8g"
server_key: "8c062f7d78513658430a3f941644eb85"
server_key_iv: "eb6b1237e3d2b7fb43d440d5cc4b94d9"
server_key_version: "encryptionkey@0"
I20260928 10:50:59.349753 9601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:50:59.386950 9601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:50:59.392948 9601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:50:59.393396 9601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:50:59.416822 9601 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:39323
I20260928 10:50:59.416822 9667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:39323 every 8 connection(s)
I20260928 10:50:59.418695 9601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:50:59.424400 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9601
I20260928 10:50:59.424417 9668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:50:59.424651 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:50:59.425550 6030 external_mini_cluster.cc:1428] Setting key a62c0557527b1c72692015be3c6ec1af
I20260928 10:50:59.441172 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163: Bootstrap starting.
Sep 28 10:50:59 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:50:59.446010 9668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163: Neither blocks nor log segments found. Creating new log.
I20260928 10:50:59.447500 9668 log.cc:824] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163: Log is configured to *not* fsync() on all Append() calls
I20260928 10:50:59.451905 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163: No bootstrap required, opened a new log
I20260928 10:50:59.457226 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:59.428381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.5.227.190:39323)
0928 10:50:59.429295 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:59.429308 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:59.429351 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:50:59.432016 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:59.432045 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:59.432553 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:59.433146 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:59.435131 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:59.437184 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:59.438659 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:59.439419 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:59.439650 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:59.446292 (+ 6642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:50:59.446301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:50:59.446319 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:50:59.446379 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:50:59.450859 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:59.452156 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:59.452197 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:59.452209 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:59.452428 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:50:59.452917 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:50:59.452921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:50:59.452923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:50:59.453727 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:50:59.454175 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:59.454720 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 10:50:59.455527 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":385,"threads_started":1}
I20260928 10:50:59.464731 9668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e9f1ccadc455b8d23d0c739187163" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39323 } }
I20260928 10:50:59.465417 9668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:50:59.465553 9668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8e9f1ccadc455b8d23d0c739187163, State: Initialized, Role: FOLLOWER
I20260928 10:50:59.466248 9668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e9f1ccadc455b8d23d0c739187163" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39323 } }
I20260928 10:50:59.466521 9668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:50:59.466678 9668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:50:59.466873 9668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:50:59.469436 9668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e9f1ccadc455b8d23d0c739187163" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39323 } }
I20260928 10:50:59.470045 9668 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8e9f1ccadc455b8d23d0c739187163; no voters:
I20260928 10:50:59.470861 9668 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:50:59.471218 9673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:50:59.471989 9673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [term 1 LEADER]: Becoming Leader. State: Replica: 8b8e9f1ccadc455b8d23d0c739187163, State: Running, Role: LEADER
I20260928 10:50:59.472886 9673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e9f1ccadc455b8d23d0c739187163" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39323 } }
I20260928 10:50:59.473289 9668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:50:59.477525 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8e9f1ccadc455b8d23d0c739187163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e9f1ccadc455b8d23d0c739187163" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39323 } } }
I20260928 10:50:59.477608 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8e9f1ccadc455b8d23d0c739187163. Latest consensus state: current_term: 1 leader_uuid: "8b8e9f1ccadc455b8d23d0c739187163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8e9f1ccadc455b8d23d0c739187163" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39323 } } }
I20260928 10:50:59.477952 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:59.477984 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163 [sys.catalog]: This master's current role is: LEADER
I20260928 10:50:59.479148 9682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:50:59.489624 9682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:50:59.500567 9682 catalog_manager.cc:1347] Generated new cluster ID: 9fc51f8100f5452abc77811cf9acaec7
I20260928 10:50:59.500718 9682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:50:59.522344 9682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:50:59.524863 9682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:50:59.538949 9682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b8e9f1ccadc455b8d23d0c739187163: Generated new TSK 0
I20260928 10:50:59.540066 9682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:50:59.602065 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39323
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:50:59.992682 9694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:50:59.993203 9694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:50:59.993400 9694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:00.002375 9694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:00.002605 9694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:00.002801 9694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:00.014128 9694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:00.015676 9694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:00.017580 9694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:00.034075 9699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:00.034052 9700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:00.034155 9702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:00.035264 9694 server_base.cc:1034] running on GCE node
I20260928 10:51:00.036206 9694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:00.037328 9694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:00.038642 9694 hybrid_clock.cc:648] HybridClock initialized: now 1790592660038568 us; error 68 us; skew 500 ppm
I20260928 10:51:00.039321 9694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:00 dist-test-slave-8w8g krb5kdc[9588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592660, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:00.044127 9694 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:51:00.046100 9694 webserver.cc:460] Webserver started at http://127.5.227.129:35409/ using document root <none> and password file <none>
I20260928 10:51:00.046866 9694 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:00.046985 9694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:00.047367 9694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:00.051463 9694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "b76be267222c4e9cb40bd3e9bdf4f440"
format_stamp: "Formatted at 2026-09-28 10:51:00 on dist-test-slave-8w8g"
server_key: "f969e1bfe8c8ce664d59a1d5469961ba"
server_key_iv: "a23f000b543dc2d4f20e26540c5a18c9"
server_key_version: "encryptionkey@0"
I20260928 10:51:00.052915 9694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "b76be267222c4e9cb40bd3e9bdf4f440"
format_stamp: "Formatted at 2026-09-28 10:51:00 on dist-test-slave-8w8g"
server_key: "f969e1bfe8c8ce664d59a1d5469961ba"
server_key_iv: "a23f000b543dc2d4f20e26540c5a18c9"
server_key_version: "encryptionkey@0"
I20260928 10:51:00.061729 9694 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260928 10:51:00.066875 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:00.068694 9694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 10:51:00.068910 9694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "b76be267222c4e9cb40bd3e9bdf4f440"
format_stamp: "Formatted at 2026-09-28 10:51:00 on dist-test-slave-8w8g"
server_key: "f969e1bfe8c8ce664d59a1d5469961ba"
server_key_iv: "a23f000b543dc2d4f20e26540c5a18c9"
server_key_version: "encryptionkey@0"
I20260928 10:51:00.069170 9694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:00.091404 9694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:00.098184 9694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:00.098898 9694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:00.100975 9694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:00.103075 9694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:00.103191 9694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:00.103298 9694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:00.103349 9694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:00.162372 9694 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:37595
I20260928 10:51:00.162456 9821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:37595 every 8 connection(s)
I20260928 10:51:00.164793 9694 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:00.171061 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9694
I20260928 10:51:00.171283 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:51:00.172437 6030 external_mini_cluster.cc:1428] Setting key d343cb95c2e2e44c67738bff6cb34b90
Sep 28 10:51:00 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592660, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:00.204706 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:00.169009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:42657 (local address 127.5.227.190:39323)
0928 10:51:00.169565 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:00.169570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:00.174936 (+ 5366us) server_negotiation.cc:407] Connection header received
0928 10:51:00.178547 (+ 3611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:00.178553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:00.178658 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:00.178800 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:00.181246 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.182717 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.185883 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.186573 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.186755 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:00.194663 (+ 7908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:00.194673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:00.194678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:00.194730 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:00.199405 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:00.200710 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:00.200717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:00.200721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:00.200883 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:00.201421 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:00.201427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:00.201431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:00.201915 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:00.202094 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:00.203445 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0928 10:51:00.203771 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":300,"threads_started":1}
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:51:00.206077 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:00.169883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39323 (local address 127.5.227.129:42657)
0928 10:51:00.174729 (+ 4846us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:00.174796 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:00.178056 (+ 3260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:00.179138 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:00.179154 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:00.180050 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:00.180963 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:00.180993 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.182990 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.183005 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:00.185642 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:00.185663 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.186940 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.187041 (+ 101us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:00.187520 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:00.189095 (+ 1575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:00.189132 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:00.194271 (+ 5139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:00.199648 (+ 5377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:00.199661 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:00.199683 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:00.200480 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:00.201028 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:00.201033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:00.201037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:00.201270 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:00.202071 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:00.202081 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:00.202522 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 10:51:00.203212 (+ 690us) client_negotiation.cc:240] Negotiation successful
0928 10:51:00.204602 (+ 1390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3695,"thread_start_us":3446,"threads_started":1}
I20260928 10:51:00.208946 9822 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39323
I20260928 10:51:00.209405 9822 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:00.210564 9822 heartbeater.cc:511] Master 127.5.227.190:39323 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:51:00.214138 9633 ts_manager.cc:194] Registered new tserver with Master: b76be267222c4e9cb40bd3e9bdf4f440 (127.5.227.129:37595)
I20260928 10:51:00.216405 9633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:42657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:51:00.258189 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39323
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:00.667357 9828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:00.667685 9828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:00.667848 9828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:00.675915 9828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:00.676044 9828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:00.676163 9828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:00.687384 9828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:00.689126 9828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:00.690984 9828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:00.708370 9836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:00.708621 9833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:00.708402 9834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:00.709620 9828 server_base.cc:1034] running on GCE node
I20260928 10:51:00.710316 9828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:00.711457 9828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:00.712749 9828 hybrid_clock.cc:648] HybridClock initialized: now 1790592660712687 us; error 88 us; skew 500 ppm
I20260928 10:51:00.713160 9828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:00 dist-test-slave-8w8g krb5kdc[9588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592660, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:00.717351 9828 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:51:00.719301 9828 webserver.cc:460] Webserver started at http://127.5.227.130:40303/ using document root <none> and password file <none>
I20260928 10:51:00.720223 9828 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:00.720397 9828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:00.720754 9828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:00.724287 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "66f491b285e94df6a1e8460b5b2e5c0a"
format_stamp: "Formatted at 2026-09-28 10:51:00 on dist-test-slave-8w8g"
server_key: "1304ee7e003cea4c6922adb4c63401d0"
server_key_iv: "94fd599ad855f63c7f4f099e299ed192"
server_key_version: "encryptionkey@0"
I20260928 10:51:00.725257 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "66f491b285e94df6a1e8460b5b2e5c0a"
format_stamp: "Formatted at 2026-09-28 10:51:00 on dist-test-slave-8w8g"
server_key: "1304ee7e003cea4c6922adb4c63401d0"
server_key_iv: "94fd599ad855f63c7f4f099e299ed192"
server_key_version: "encryptionkey@0"
I20260928 10:51:00.732199 9828 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 10:51:00.737844 9843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:00.739931 9828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260928 10:51:00.740119 9828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "66f491b285e94df6a1e8460b5b2e5c0a"
format_stamp: "Formatted at 2026-09-28 10:51:00 on dist-test-slave-8w8g"
server_key: "1304ee7e003cea4c6922adb4c63401d0"
server_key_iv: "94fd599ad855f63c7f4f099e299ed192"
server_key_version: "encryptionkey@0"
I20260928 10:51:00.740331 9828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:00.776458 9828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:00.781611 9828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:00.781894 9828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:00.782881 9828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:00.784453 9828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:00.784533 9828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:00.784637 9828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:00.784674 9828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:00.823539 9828 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:41431
I20260928 10:51:00.823561 9955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:41431 every 8 connection(s)
I20260928 10:51:00.825407 9828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:00.834998 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9828
I20260928 10:51:00.835180 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:51:00.835770 6030 external_mini_cluster.cc:1428] Setting key 392ec4542a16c0664308879eec1e2bfa
Sep 28 10:51:00 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592660, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:00.852573 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:00.828664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:52677 (local address 127.5.227.190:39323)
0928 10:51:00.829244 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:00.829256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:00.830491 (+ 1235us) server_negotiation.cc:407] Connection header received
0928 10:51:00.832595 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:00.832600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:00.832682 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:00.832797 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:00.834624 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.835846 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.838238 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.839139 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.839314 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:00.844435 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:00.844445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:00.844450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:00.844529 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:00.848613 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:00.849544 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:00.849554 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:00.849560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:00.849741 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:00.850346 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:00.850355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:00.850361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:00.850786 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:00.850951 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:00.851795 (+ 844us) server_negotiation.cc:299] Negotiation successful
0928 10:51:00.852065 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":219,"threads_started":1}
I20260928 10:51:00.853752 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:00.829474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39323 (local address 127.5.227.130:52677)
0928 10:51:00.830309 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:00.830354 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:00.832258 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:00.833092 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:00.833101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:00.833722 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:00.834381 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:00.834401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.836201 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.836206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:00.838015 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:00.838034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:00.839444 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:00.839452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:00.839708 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:00.840611 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:00.840642 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:00.844091 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:00.848813 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:00.848822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:00.848838 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:00.849389 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:00.849950 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:00.849956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:00.849967 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:00.850192 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:00.850960 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:00.850967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:00.851355 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 10:51:00.851630 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 10:51:00.852251 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":328,"threads_started":1}
I20260928 10:51:00.855719 9956 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39323
I20260928 10:51:00.856226 9956 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:00.857139 9956 heartbeater.cc:511] Master 127.5.227.190:39323 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:51:00.859457 9633 ts_manager.cc:194] Registered new tserver with Master: 66f491b285e94df6a1e8460b5b2e5c0a (127.5.227.130:41431)
I20260928 10:51:00.860845 9633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:52677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:51:00.897548 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39323
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:01.221078 9822 heartbeater.cc:503] Master 127.5.227.190:39323 was elected leader, sending a full tablet report...
W20260928 10:51:01.303638 9962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:01.304077 9962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:01.304246 9962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:01.312829 9962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:01.313133 9962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:01.313352 9962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:01.324457 9962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:01.326383 9962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:01.328027 9962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:01.346709 9967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:01.346876 9968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:01.346728 9970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:01.347307 9962 server_base.cc:1034] running on GCE node
I20260928 10:51:01.348515 9962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:01.349640 9962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:01.350932 9962 hybrid_clock.cc:648] HybridClock initialized: now 1790592661350832 us; error 73 us; skew 500 ppm
I20260928 10:51:01.351414 9962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:01 dist-test-slave-8w8g krb5kdc[9588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:01.356212 9962 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:51:01.358225 9962 webserver.cc:460] Webserver started at http://127.5.227.131:36135/ using document root <none> and password file <none>
I20260928 10:51:01.360096 9962 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:01.360376 9962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:01.360796 9962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:01.365185 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "97cd55e4a5904d5495eed7606bbf6b00"
format_stamp: "Formatted at 2026-09-28 10:51:01 on dist-test-slave-8w8g"
server_key: "23b16801ec305e1ef168963992b307eb"
server_key_iv: "60998e5ff3b43fdb29b57976fdb43cd8"
server_key_version: "encryptionkey@0"
I20260928 10:51:01.366676 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "97cd55e4a5904d5495eed7606bbf6b00"
format_stamp: "Formatted at 2026-09-28 10:51:01 on dist-test-slave-8w8g"
server_key: "23b16801ec305e1ef168963992b307eb"
server_key_iv: "60998e5ff3b43fdb29b57976fdb43cd8"
server_key_version: "encryptionkey@0"
I20260928 10:51:01.373641 9962 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:51:01.378789 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:01.380774 9962 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:51:01.381006 9962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "97cd55e4a5904d5495eed7606bbf6b00"
format_stamp: "Formatted at 2026-09-28 10:51:01 on dist-test-slave-8w8g"
server_key: "23b16801ec305e1ef168963992b307eb"
server_key_iv: "60998e5ff3b43fdb29b57976fdb43cd8"
server_key_version: "encryptionkey@0"
I20260928 10:51:01.381292 9962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:01.421329 9962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:01.427476 9962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:01.427891 9962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:01.429080 9962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:01.431061 9962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:01.431169 9962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:01.431326 9962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:01.431393 9962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:01.479211 9962 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:37357
I20260928 10:51:01.479221 10089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:37357 every 8 connection(s)
I20260928 10:51:01.481284 9962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:01.485852 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 9962
I20260928 10:51:01.486331 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:51:01.486905 6030 external_mini_cluster.cc:1428] Setting key 099b422bc61a7434db42bc13b8992dc1
Sep 28 10:51:01 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:01.512179 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:01.485564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:37491 (local address 127.5.227.190:39323)
0928 10:51:01.486353 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:01.486359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:01.488413 (+ 2054us) server_negotiation.cc:407] Connection header received
0928 10:51:01.491231 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:01.491236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:01.491303 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:01.491551 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:01.494199 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.495206 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.497708 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.498354 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.498495 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:01.503212 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:01.503219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:01.503223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:01.503270 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:01.507176 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.508556 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.508561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.508564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.508717 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.509447 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.509454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.509458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.509871 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:01.510117 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:01.511235 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0928 10:51:01.511662 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":276,"threads_started":1}
I20260928 10:51:01.513844 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:01.486615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39323 (local address 127.5.227.131:37491)
0928 10:51:01.488177 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:01.488270 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:01.490826 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:01.492338 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:01.492349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:01.493175 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:01.493845 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:01.493867 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.495421 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.495425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:01.497406 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:01.497425 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.498665 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.498671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:01.498892 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:01.499795 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:01.499828 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:01.502946 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:01.507422 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:01.507435 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:01.507467 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:01.508142 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:01.508862 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:01.508868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:01.508872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:01.509259 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:01.510054 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:01.510066 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:01.510650 (+ 584us) client_negotiation.cc:769] Sending connection context
0928 10:51:01.510981 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 10:51:01.511996 (+ 1015us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":906,"threads_started":1}
I20260928 10:51:01.516767 10090 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39323
I20260928 10:51:01.517215 10090 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:01.518271 10090 heartbeater.cc:511] Master 127.5.227.190:39323 requested a full tablet report, sending...
I20260928 10:51:01.520460 9633 ts_manager.cc:194] Registered new tserver with Master: 97cd55e4a5904d5495eed7606bbf6b00 (127.5.227.131:37357)
I20260928 10:51:01.521363 9633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:37491
I20260928 10:51:01.533762 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:51:01 dist-test-slave-8w8g krb5kdc[9588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:51:01 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:51:01.608659 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:01.581015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.5.227.129:37595)
0928 10:51:01.586573 (+ 5558us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:01.586582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:01.586619 (+ 37us) server_negotiation.cc:407] Connection header received
0928 10:51:01.586695 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:01.586702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:01.587293 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:01.587532 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:01.589917 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.591145 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.593138 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.594069 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.594493 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:01.599708 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:01.599725 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:01.599918 (+ 193us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:01.599980 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:01.603910 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.604936 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.604946 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.604949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.605104 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.605732 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.605737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.605739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.606355 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:01.606692 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:01.607495 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 10:51:01.608120 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5338,"thread_start_us":5110,"threads_started":1}
Sep 28 10:51:01 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:01.636407 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:01.615374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.5.227.190:39323)
0928 10:51:01.615607 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:01.615614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:01.615895 (+ 281us) server_negotiation.cc:407] Connection header received
0928 10:51:01.616128 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:01.616134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:01.616252 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:01.616372 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:01.618107 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.618862 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.621195 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.621661 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.622279 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:01.627785 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:01.627796 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:01.627803 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:01.627912 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:01.631710 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.632796 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.632806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.632808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.632992 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.633862 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.633874 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.633878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.634280 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:01.634781 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:01.635447 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 10:51:01.635791 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 10:51:01.637077 9633 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43388
I20260928 10:51:01.638623 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9694
I20260928 10:51:01.655983 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:01.639846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.5.227.129:37595)
0928 10:51:01.640241 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:01.640248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:01.640374 (+ 126us) server_negotiation.cc:407] Connection header received
0928 10:51:01.640617 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:01.640622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:01.640690 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:01.640802 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:01.642168 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.642862 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.644323 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.644799 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.644948 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:01.648893 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:01.648900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:01.648904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:01.648958 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:01.652239 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.653540 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.653547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.653550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.653786 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.654379 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.654384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.654388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.654702 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:01.654983 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:01.655344 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 10:51:01.655597 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 10:51:01.656697 9817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:01.752606 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9694
I20260928 10:51:01.772207 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9828
Sep 28 10:51:01 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:51:01.795102 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:01.773513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.5.227.130:41431)
0928 10:51:01.774270 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:01.774287 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:01.774316 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:51:01.774411 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:01.774417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:01.774641 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:01.774789 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:01.776819 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.777925 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.779550 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.780452 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.780593 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:01.786423 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:01.786450 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:01.786467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:01.786523 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:01.790613 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.792168 (+ 1555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.792177 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.792180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.792343 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.792899 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.792904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.792906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.793307 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:01.793671 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:01.794035 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 10:51:01.794329 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":339,"threads_started":1}
I20260928 10:51:01.797224 9951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:01.895149 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9828
I20260928 10:51:01.913776 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9962
Sep 28 10:51:01 dist-test-slave-8w8g krb5kdc[9588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:51:01.934904 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:01.914996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.5.227.131:37357)
0928 10:51:01.915584 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:01.915592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:01.915629 (+ 37us) server_negotiation.cc:407] Connection header received
0928 10:51:01.915690 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:01.915696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:01.916002 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:01.916155 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:01.917576 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.918628 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.920555 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:01.921177 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:01.921299 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:01.927084 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:01.927104 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:01.927123 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:01.927172 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:01.930765 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.931885 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.931895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.931897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.932103 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:01.932924 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:01.932930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:01.932934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:01.933387 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:01.933728 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:01.934311 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 10:51:01.934567 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260928 10:51:01.936686 10085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:02.032346 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9962
I20260928 10:51:02.049714 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 9601
I20260928 10:51:02.051226 9663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:02.160888 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 9601
2026-09-28T10:51:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4683 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:51:02 dist-test-slave-8w8g krb5kdc[10110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:51:02 dist-test-slave-8w8g krb5kdc[10110](info): set up 2 sockets
Sep 28 10:51:02 dist-test-slave-8w8g krb5kdc[10110](info): commencing operation
krb5kdc: starting...
W20260928 10:51:03.409337 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:51:03 dist-test-slave-8w8g krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:51:03.673702 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37365
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:37365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:04.056627 10123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:04.057034 10123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:04.057215 10123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:04.066434 10123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:04.066643 10123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:04.066764 10123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:04.066872 10123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:04.066982 10123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:04.078140 10123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:37365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:04.079710 10123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:04.081290 10123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:04.096338 10128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:04.096379 10129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:04.096413 10131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:04.097158 10123 server_base.cc:1034] running on GCE node
I20260928 10:51:04.098685 10123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:04.101171 10123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:04.102452 10123 hybrid_clock.cc:648] HybridClock initialized: now 1790592664102375 us; error 78 us; skew 500 ppm
I20260928 10:51:04.102927 10123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:04 dist-test-slave-8w8g krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:04.107059 10123 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:51:04.108711 10123 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:04.109261 10123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:04.109730 10123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:04.113646 10123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "6f7372d0fcfe47e086901b1738d2da49"
format_stamp: "Formatted at 2026-09-28 10:51:04 on dist-test-slave-8w8g"
server_key: "80c386914a1b323ef079aea3b7efc35d"
server_key_iv: "ccdb0dc1ce0712ccb205e9f875e36e8b"
server_key_version: "encryptionkey@0"
I20260928 10:51:04.114675 10123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "6f7372d0fcfe47e086901b1738d2da49"
format_stamp: "Formatted at 2026-09-28 10:51:04 on dist-test-slave-8w8g"
server_key: "80c386914a1b323ef079aea3b7efc35d"
server_key_iv: "ccdb0dc1ce0712ccb205e9f875e36e8b"
server_key_version: "encryptionkey@0"
I20260928 10:51:04.121533 10123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:51:04.126415 10137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:04.128373 10123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:51:04.128602 10123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "6f7372d0fcfe47e086901b1738d2da49"
format_stamp: "Formatted at 2026-09-28 10:51:04 on dist-test-slave-8w8g"
server_key: "80c386914a1b323ef079aea3b7efc35d"
server_key_iv: "ccdb0dc1ce0712ccb205e9f875e36e8b"
server_key_version: "encryptionkey@0"
I20260928 10:51:04.128886 10123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:04.159772 10123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:04.166028 10123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:04.166347 10123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:04.194810 10123 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:37365
I20260928 10:51:04.194839 10188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:37365 every 8 connection(s)
I20260928 10:51:04.196669 10123 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:04.201588 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 10123
I20260928 10:51:04.201856 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:04.202598 10189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:04.202845 6030 external_mini_cluster.cc:1428] Setting key aae9acbb60311814da5384899dc5e977
I20260928 10:51:04.221366 10189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49: Bootstrap starting.
Sep 28 10:51:04 dist-test-slave-8w8g krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:04.225493 10189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:04.227085 10189 log.cc:824] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:04.230784 10189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49: No bootstrap required, opened a new log
I20260928 10:51:04.234159 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:04.205617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.5.227.190:37365)
0928 10:51:04.206571 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:04.206585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:04.206633 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:51:04.208882 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:04.208916 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:04.209436 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:04.209879 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:04.212528 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:04.214010 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:04.216519 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:04.217598 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:04.217784 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:04.224863 (+ 7079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:04.224874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:04.224898 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:04.224958 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:04.229062 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:04.230016 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:04.230023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:04.230031 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:04.230214 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:04.230662 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:04.230667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:04.230669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:04.231329 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:04.231698 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:04.232244 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 10:51:04.232726 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":438,"threads_started":1}
I20260928 10:51:04.243168 10189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7372d0fcfe47e086901b1738d2da49" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37365 } }
I20260928 10:51:04.244148 10189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:04.244314 10189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7372d0fcfe47e086901b1738d2da49, State: Initialized, Role: FOLLOWER
I20260928 10:51:04.245118 10189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7372d0fcfe47e086901b1738d2da49" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37365 } }
I20260928 10:51:04.245404 10189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:04.245565 10189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:04.245755 10189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:04.248875 10189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7372d0fcfe47e086901b1738d2da49" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37365 } }
I20260928 10:51:04.249542 10189 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7372d0fcfe47e086901b1738d2da49; no voters:
I20260928 10:51:04.250244 10189 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:04.250511 10194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:04.251441 10194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [term 1 LEADER]: Becoming Leader. State: Replica: 6f7372d0fcfe47e086901b1738d2da49, State: Running, Role: LEADER
I20260928 10:51:04.252180 10194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7372d0fcfe47e086901b1738d2da49" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37365 } }
I20260928 10:51:04.253412 10189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:04.256062 10195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7372d0fcfe47e086901b1738d2da49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7372d0fcfe47e086901b1738d2da49" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37365 } } }
I20260928 10:51:04.257115 10195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:04.256685 10196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7372d0fcfe47e086901b1738d2da49. Latest consensus state: current_term: 1 leader_uuid: "6f7372d0fcfe47e086901b1738d2da49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7372d0fcfe47e086901b1738d2da49" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37365 } } }
I20260928 10:51:04.257236 10196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:04.258121 10202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:04.271210 10202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:04.285575 10202 catalog_manager.cc:1347] Generated new cluster ID: 8b51f711bef2409daa9947f1908976ac
I20260928 10:51:04.285931 10202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:04.333580 10202 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:04.335703 10202 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:04.352182 10202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49: Generated new TSK 0
I20260928 10:51:04.353605 10202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:51:04.427201 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37365
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:04.830344 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:04.830685 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:04.830878 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:04.839840 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:04.839998 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:04.840121 10215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:04.852801 10215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:04.855088 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:04.857355 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:04.876153 10220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:04.876127 10223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:04.876127 10221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:04.877852 10215 server_base.cc:1034] running on GCE node
I20260928 10:51:04.878621 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:04.880415 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:04.881722 10215 hybrid_clock.cc:648] HybridClock initialized: now 1790592664881663 us; error 40 us; skew 500 ppm
I20260928 10:51:04.882207 10215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:04 dist-test-slave-8w8g krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:04.894454 10215 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:51:04.896308 10215 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:04.896492 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:04.896872 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:04.900482 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "3447fe184f0547bb96a29fb3851c5b45"
format_stamp: "Formatted at 2026-09-28 10:51:04 on dist-test-slave-8w8g"
server_key: "2d21ce8cca92f00dac305114ed582f10"
server_key_iv: "df4b2cf316502fe92b584a4b7bbbf94d"
server_key_version: "encryptionkey@0"
I20260928 10:51:04.901477 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "3447fe184f0547bb96a29fb3851c5b45"
format_stamp: "Formatted at 2026-09-28 10:51:04 on dist-test-slave-8w8g"
server_key: "2d21ce8cca92f00dac305114ed582f10"
server_key_iv: "df4b2cf316502fe92b584a4b7bbbf94d"
server_key_version: "encryptionkey@0"
I20260928 10:51:04.907495 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:51:04.912704 10229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:04.914577 10215 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:51:04.914841 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "3447fe184f0547bb96a29fb3851c5b45"
format_stamp: "Formatted at 2026-09-28 10:51:04 on dist-test-slave-8w8g"
server_key: "2d21ce8cca92f00dac305114ed582f10"
server_key_iv: "df4b2cf316502fe92b584a4b7bbbf94d"
server_key_version: "encryptionkey@0"
I20260928 10:51:04.915253 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:04.930230 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:04.936779 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:04.937140 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:04.938328 10215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:04.940044 10215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:04.940178 10215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:04.940326 10215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:04.940428 10215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:04.990630 10215 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:43815
I20260928 10:51:04.990686 10341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:43815 every 8 connection(s)
I20260928 10:51:04.991602 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:04.995466 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 10215
I20260928 10:51:04.995690 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:51:04.996592 6030 external_mini_cluster.cc:1428] Setting key 070be4a6e0b8da27861a7b3ec772053a
Sep 28 10:51:05 dist-test-slave-8w8g krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:05.020804 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:04.995033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:38649 (local address 127.5.227.190:37365)
0928 10:51:04.995639 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:04.995648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:04.997116 (+ 1468us) server_negotiation.cc:407] Connection header received
0928 10:51:04.999337 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:04.999341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:04.999448 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:04.999575 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:05.001526 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.002741 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.005445 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.006407 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.006613 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:05.011506 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:05.011518 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:05.011526 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:05.011583 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:05.016084 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:05.017247 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:05.017257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:05.017262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:05.017470 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:05.018101 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:05.018110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:05.018116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:05.018594 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:05.018773 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:05.019900 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0928 10:51:05.020292 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":258,"threads_started":1}
I20260928 10:51:05.021670 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:04.995856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37365 (local address 127.5.227.129:38649)
0928 10:51:04.996886 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:04.996939 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:04.998950 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:04.999899 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:04.999909 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:05.000604 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:05.001300 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:05.001320 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.003057 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.003062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:05.005175 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:05.005202 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.006770 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.006776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:05.007053 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:05.008090 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:05.008126 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:05.011058 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:05.016377 (+ 5319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:05.016387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:05.016406 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:05.017041 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:05.017664 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:05.017671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:05.017673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:05.017932 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:05.018789 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:05.018797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:05.019337 (+ 540us) client_negotiation.cc:769] Sending connection context
0928 10:51:05.019628 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 10:51:05.020220 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":398,"threads_started":1}
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:51:05.023698 10342 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37365
I20260928 10:51:05.024255 10342 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:05.025221 10342 heartbeater.cc:511] Master 127.5.227.190:37365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:51:05.028671 10154 ts_manager.cc:194] Registered new tserver with Master: 3447fe184f0547bb96a29fb3851c5b45 (127.5.227.129:43815)
I20260928 10:51:05.030797 10154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:38649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:51:05.065522 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37365
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:05.473526 10348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:05.473886 10348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:05.474067 10348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:05.482999 10348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:05.483157 10348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:05.483387 10348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:05.494055 10348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:05.495569 10348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:05.497001 10348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:05.512095 10353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:05.512079 10354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:05.512494 10356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:05.513146 10348 server_base.cc:1034] running on GCE node
I20260928 10:51:05.513881 10348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:05.515116 10348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:05.516494 10348 hybrid_clock.cc:648] HybridClock initialized: now 1790592665516366 us; error 59 us; skew 500 ppm
I20260928 10:51:05.516971 10348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:05 dist-test-slave-8w8g krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592665, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:05.520815 10348 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:51:05.522320 10348 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:05.522476 10348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:05.522821 10348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:05.526417 10348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "3fe9910c4d5b4304bafe35037caafa79"
format_stamp: "Formatted at 2026-09-28 10:51:05 on dist-test-slave-8w8g"
server_key: "d5b8f8d7d2cbcf33db284f826938aa18"
server_key_iv: "34886df893723fec05bfbddc10a8819c"
server_key_version: "encryptionkey@0"
I20260928 10:51:05.527462 10348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "3fe9910c4d5b4304bafe35037caafa79"
format_stamp: "Formatted at 2026-09-28 10:51:05 on dist-test-slave-8w8g"
server_key: "d5b8f8d7d2cbcf33db284f826938aa18"
server_key_iv: "34886df893723fec05bfbddc10a8819c"
server_key_version: "encryptionkey@0"
I20260928 10:51:05.533532 10348 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 10:51:05.538372 10362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:05.540489 10348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:51:05.540702 10348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "3fe9910c4d5b4304bafe35037caafa79"
format_stamp: "Formatted at 2026-09-28 10:51:05 on dist-test-slave-8w8g"
server_key: "d5b8f8d7d2cbcf33db284f826938aa18"
server_key_iv: "34886df893723fec05bfbddc10a8819c"
server_key_version: "encryptionkey@0"
I20260928 10:51:05.540998 10348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:05.563868 10348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:05.569885 10348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:05.570314 10348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:05.572395 10348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:05.575044 10348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:05.575182 10348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:05.575320 10348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:05.575448 10348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 10:51:05.616969 10348 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:44379
I20260928 10:51:05.617168 10474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:44379 every 8 connection(s)
I20260928 10:51:05.617951 10348 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:05.622423 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 10348
I20260928 10:51:05.622630 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:51:05.623435 6030 external_mini_cluster.cc:1428] Setting key ff92d2fdf8e1e519f10265a843128032
Sep 28 10:51:05 dist-test-slave-8w8g krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592665, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:05.644160 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:05.621177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:38841 (local address 127.5.227.190:37365)
0928 10:51:05.621721 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:05.621728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:05.623080 (+ 1352us) server_negotiation.cc:407] Connection header received
0928 10:51:05.625647 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:05.625653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:05.625740 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:05.626131 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:05.628378 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.629136 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.630908 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.631382 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.631542 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:05.636170 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:05.636180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:05.636188 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:05.636242 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:05.640399 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:05.641530 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:05.641536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:05.641538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:05.641671 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:05.642176 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:05.642186 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:05.642188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:05.642480 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:05.642671 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:05.643462 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 10:51:05.643715 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":284,"threads_started":1}
I20260928 10:51:05.645465 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:05.621722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37365 (local address 127.5.227.130:38841)
0928 10:51:05.622893 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:05.622935 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:05.625292 (+ 2357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:05.626425 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:05.626436 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:05.627296 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:05.628131 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:05.628156 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.629349 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.629353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:05.630694 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:05.630711 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:05.631663 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:05.631670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:05.631975 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:05.632880 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:05.632906 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:05.635845 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:05.640759 (+ 4914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:05.640768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:05.640785 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:05.641335 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:05.641809 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:05.641814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:05.641819 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:05.642034 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:05.642655 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:05.642664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:05.643064 (+ 400us) client_negotiation.cc:769] Sending connection context
0928 10:51:05.643373 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 10:51:05.644029 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":572,"threads_started":1}
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:51:05.647872 10475 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37365
I20260928 10:51:05.648306 10475 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:05.649129 10475 heartbeater.cc:511] Master 127.5.227.190:37365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:51:05.651706 10154 ts_manager.cc:194] Registered new tserver with Master: 3fe9910c4d5b4304bafe35037caafa79 (127.5.227.130:44379)
I20260928 10:51:05.653540 10154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:38841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:51:05.689734 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37365
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:06.035367 10342 heartbeater.cc:503] Master 127.5.227.190:37365 was elected leader, sending a full tablet report...
W20260928 10:51:06.107998 10481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:06.108322 10481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:06.108479 10481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:06.117383 10481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:06.117708 10481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:06.118130 10481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:06.130815 10481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:06.132572 10481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:06.135000 10481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:06.154227 10486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:06.154542 10489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:06.154783 10487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:06.156149 10481 server_base.cc:1034] running on GCE node
I20260928 10:51:06.156978 10481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:06.158185 10481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:06.159512 10481 hybrid_clock.cc:648] HybridClock initialized: now 1790592666159462 us; error 42 us; skew 500 ppm
I20260928 10:51:06.159991 10481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:06 dist-test-slave-8w8g krb5kdc[10110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592666, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:06.164633 10481 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:51:06.166733 10481 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:06.166994 10481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:06.167335 10481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:06.170804 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "cf989722d97442f7b8c79c199038a7ff"
format_stamp: "Formatted at 2026-09-28 10:51:06 on dist-test-slave-8w8g"
server_key: "daf7c5c68e4fc894c20ae7d628cdf3be"
server_key_iv: "c9c811e728f94628ed77f484a3ab9c08"
server_key_version: "encryptionkey@0"
I20260928 10:51:06.171826 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "cf989722d97442f7b8c79c199038a7ff"
format_stamp: "Formatted at 2026-09-28 10:51:06 on dist-test-slave-8w8g"
server_key: "daf7c5c68e4fc894c20ae7d628cdf3be"
server_key_iv: "c9c811e728f94628ed77f484a3ab9c08"
server_key_version: "encryptionkey@0"
I20260928 10:51:06.179299 10481 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 10:51:06.184593 10495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:06.186707 10481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 10:51:06.186957 10481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "cf989722d97442f7b8c79c199038a7ff"
format_stamp: "Formatted at 2026-09-28 10:51:06 on dist-test-slave-8w8g"
server_key: "daf7c5c68e4fc894c20ae7d628cdf3be"
server_key_iv: "c9c811e728f94628ed77f484a3ab9c08"
server_key_version: "encryptionkey@0"
I20260928 10:51:06.187233 10481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:06.212137 10481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:06.217726 10481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:06.218103 10481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:06.219316 10481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:06.221237 10481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:06.221343 10481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:06.221472 10481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:06.221547 10481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:06.269369 10481 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:39795
I20260928 10:51:06.269448 10607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:39795 every 8 connection(s)
I20260928 10:51:06.270741 10481 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:06.278354 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 10481
I20260928 10:51:06.278564 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:51:06.279088 6030 external_mini_cluster.cc:1428] Setting key f0ddefeca465e2bee820cdfc02e7d994
Sep 28 10:51:06 dist-test-slave-8w8g krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592666, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:06.296128 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.273820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:57237 (local address 127.5.227.190:37365)
0928 10:51:06.274559 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.274564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.275639 (+ 1075us) server_negotiation.cc:407] Connection header received
0928 10:51:06.277675 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.277680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.277786 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.277937 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:06.280035 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.280862 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.282620 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.283029 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.283176 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.288081 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:06.288087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:06.288090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:06.288130 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:06.291347 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.292566 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.292571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.292574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.292889 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.293615 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.293619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.293621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.294148 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:06.294303 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.295541 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.295835 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":305,"threads_started":1}
I20260928 10:51:06.297243 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.274611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37365 (local address 127.5.227.131:57237)
0928 10:51:06.275433 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:06.275488 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:06.277323 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:06.278247 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:06.278258 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:06.278972 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:06.279799 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.279842 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.281062 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.281067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.282448 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.282463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.283292 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.283300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.283683 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.284561 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:06.284588 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:06.287720 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:06.291594 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:06.291607 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:06.291634 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:06.292267 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:06.293161 (+ 894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:06.293167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:06.293172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:06.293461 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:06.294351 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:06.294363 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:06.295030 (+ 667us) client_negotiation.cc:769] Sending connection context
0928 10:51:06.295407 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 10:51:06.296026 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":375,"threads_started":1}
I20260928 10:51:06.299268 10608 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37365
I20260928 10:51:06.299693 10608 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:06.300705 10608 heartbeater.cc:511] Master 127.5.227.190:37365 requested a full tablet report, sending...
I20260928 10:51:06.303189 10154 ts_manager.cc:194] Registered new tserver with Master: cf989722d97442f7b8c79c199038a7ff (127.5.227.131:39795)
I20260928 10:51:06.304276 10154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:57237
I20260928 10:51:06.309129 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:51:06.335229 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.315517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.5.227.190:37365)
0928 10:51:06.315829 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.315834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.316644 (+ 810us) server_negotiation.cc:407] Connection header received
0928 10:51:06.316865 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.316870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.316961 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.317062 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:06.319097 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.320470 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.322302 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.322976 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.323101 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.327213 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:06.327221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:06.327224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:06.327269 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:06.331016 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.332065 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.332070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.332072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.332232 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.332835 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.332841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.332983 (+ 142us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.333360 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:06.333637 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.334566 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.334777 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 10:51:06.340741 10154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:51:06.349466 10154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:06.386402 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.364679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.5.227.131:39795)
0928 10:51:06.372527 (+ 7848us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.372545 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.372620 (+ 75us) server_negotiation.cc:407] Connection header received
0928 10:51:06.372804 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.372818 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.373285 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.377411 (+ 4126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:06.377906 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.379765 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.382706 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.384431 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.385079 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.385457 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.385556 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.385750 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":1152,"threads_started":1}
I20260928 10:51:06.386492 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.364283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:39795 (local address 127.0.0.1:58070)
0928 10:51:06.365263 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:06.365310 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:06.365522 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:06.375961 (+ 10439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:06.375977 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:06.376051 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:06.376711 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.376733 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.379969 (+ 3236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.379975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.382488 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.382511 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.384698 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.384708 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.385211 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.385281 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 10:51:06.385477 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 10:51:06.385744 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":137,"thread_start_us":354,"threads_started":1}
I20260928 10:51:06.392071 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.366602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.5.227.129:43815)
0928 10:51:06.376394 (+ 9792us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.376409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.380296 (+ 3887us) server_negotiation.cc:407] Connection header received
0928 10:51:06.380557 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.380587 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.381032 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.381303 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:06.382110 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.384090 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.387979 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.389920 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.390060 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.390335 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.391355 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.391574 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9469,"thread_start_us":10297,"threads_started":1}
I20260928 10:51:06.392103 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.366314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:43815 (local address 127.0.0.1:56724)
0928 10:51:06.380114 (+ 13800us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:06.380136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:06.380311 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:06.381308 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:06.381314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:06.381341 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:06.381824 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.381839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.384278 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.384283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.387592 (+ 3309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.387622 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.390587 (+ 2965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.390598 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.391101 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.391137 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:51:06.391293 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 10:51:06.391451 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13469,"mutex_wait_us":46,"thread_start_us":12330,"threads_started":1}
I20260928 10:51:06.392650 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.365966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44379 (local address 127.0.0.1:42394)
0928 10:51:06.379142 (+ 13176us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:06.379172 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:06.379373 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:06.381591 (+ 2218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:06.381595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:06.381621 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:06.382331 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.382347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.384607 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.384612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.387644 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.387654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.390625 (+ 2971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.390632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.391176 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.391236 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 10:51:06.391347 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 10:51:06.391494 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12946,"mutex_wait_us":392,"thread_start_us":14395,"threads_started":1}
I20260928 10:51:06.392693 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.366519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.5.227.130:44379)
0928 10:51:06.380422 (+ 13903us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.380434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.380465 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:51:06.380558 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.380567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.381027 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.381303 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:06.382865 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.384427 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.387999 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.390327 (+ 2328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.390487 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.390711 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.392159 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.392365 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13726,"thread_start_us":13218,"threads_started":1}
I20260928 10:51:06.392899 10543 tablet_service.cc:1467] Processing CreateTablet for tablet c725144aab1a4f6283d124b419fa42df (DEFAULT_TABLE table=test-table [id=8f8e62f6374a43a5817e2d7229980cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:06.395354 10543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725144aab1a4f6283d124b419fa42df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:06.397733 10277 tablet_service.cc:1467] Processing CreateTablet for tablet c725144aab1a4f6283d124b419fa42df (DEFAULT_TABLE table=test-table [id=8f8e62f6374a43a5817e2d7229980cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:06.397863 10410 tablet_service.cc:1467] Processing CreateTablet for tablet c725144aab1a4f6283d124b419fa42df (DEFAULT_TABLE table=test-table [id=8f8e62f6374a43a5817e2d7229980cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:06.399189 10277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725144aab1a4f6283d124b419fa42df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:06.399377 10410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725144aab1a4f6283d124b419fa42df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:06.413057 10627 tablet_bootstrap.cc:492] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Bootstrap starting.
I20260928 10:51:06.416740 10628 tablet_bootstrap.cc:492] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Bootstrap starting.
I20260928 10:51:06.416899 10627 tablet_bootstrap.cc:654] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:06.418582 10627 log.cc:824] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:06.421969 10628 tablet_bootstrap.cc:654] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:06.423883 10629 tablet_bootstrap.cc:492] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Bootstrap starting.
I20260928 10:51:06.424436 10628 log.cc:824] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:06.428144 10629 tablet_bootstrap.cc:654] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:06.429034 10627 tablet_bootstrap.cc:492] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: No bootstrap required, opened a new log
I20260928 10:51:06.429034 10628 tablet_bootstrap.cc:492] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: No bootstrap required, opened a new log
I20260928 10:51:06.429363 10627 ts_tablet_manager.cc:1397] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.011s
I20260928 10:51:06.429363 10628 ts_tablet_manager.cc:1397] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260928 10:51:06.429829 10629 log.cc:824] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:06.435405 10629 tablet_bootstrap.cc:492] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: No bootstrap required, opened a new log
I20260928 10:51:06.435817 10629 ts_tablet_manager.cc:1397] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260928 10:51:06.441744 10628 raft_consensus.cc:348] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.441744 10627 raft_consensus.cc:348] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.442294 10628 raft_consensus.cc:374] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:06.442294 10627 raft_consensus.cc:374] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:06.442474 10627 raft_consensus.cc:729] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3447fe184f0547bb96a29fb3851c5b45, State: Initialized, Role: FOLLOWER
I20260928 10:51:06.442509 10628 raft_consensus.cc:729] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf989722d97442f7b8c79c199038a7ff, State: Initialized, Role: FOLLOWER
I20260928 10:51:06.443256 10628 consensus_queue.cc:260] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.443255 10627 consensus_queue.cc:260] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.445384 10629 raft_consensus.cc:348] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.446060 10629 raft_consensus.cc:374] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:06.446226 10629 raft_consensus.cc:729] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe9910c4d5b4304bafe35037caafa79, State: Initialized, Role: FOLLOWER
I20260928 10:51:06.446717 10608 heartbeater.cc:503] Master 127.5.227.190:37365 was elected leader, sending a full tablet report...
I20260928 10:51:06.446949 10627 ts_tablet_manager.cc:1428] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260928 10:51:06.448067 10629 consensus_queue.cc:260] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.448273 10628 ts_tablet_manager.cc:1428] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260928 10:51:06.449586 10475 heartbeater.cc:503] Master 127.5.227.190:37365 was elected leader, sending a full tablet report...
I20260928 10:51:06.450706 10629 ts_tablet_manager.cc:1428] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
I20260928 10:51:06.484095 10634 raft_consensus.cc:482] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:06.484383 10634 raft_consensus.cc:504] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.486081 10634 leader_election.cc:288] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fe9910c4d5b4304bafe35037caafa79 (127.5.227.130:44379), 3447fe184f0547bb96a29fb3851c5b45 (127.5.227.129:43815)
I20260928 10:51:06.493609 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.486697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:35885 (local address 127.5.227.129:43815)
0928 10:51:06.486897 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.486903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.486926 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:51:06.487265 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.487272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.487354 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.487474 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:06.488380 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.489306 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.491276 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.492335 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.492534 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.492632 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.493108 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.493285 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:51:06.493602 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.486430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:43815 (local address 127.5.227.131:35885)
0928 10:51:06.486808 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:06.486832 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:06.487070 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:06.487592 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:06.487600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:06.487759 (+ 159us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:06.488044 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.488059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.489477 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.489484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.491082 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.491097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.492572 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.492581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.492832 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.492875 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:51:06.492997 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 10:51:06.493166 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":1,"spinlock_wait_cycles":47488}
I20260928 10:51:06.494524 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.486696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:36447 (local address 127.5.227.130:44379)
0928 10:51:06.486895 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.486902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.487639 (+ 737us) server_negotiation.cc:407] Connection header received
0928 10:51:06.487811 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.487816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.487896 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.488010 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:06.488823 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.489716 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.492045 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.493256 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.493430 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.493499 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.494160 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.494302 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:51:06.494673 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.486655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44379 (local address 127.5.227.131:36447)
0928 10:51:06.487469 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:06.487486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:06.487653 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:06.488051 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:06.488054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:06.488075 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:06.488586 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.488598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.489945 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.489949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.491530 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:06.491547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.493655 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.493662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:06.493852 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.493886 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:51:06.494092 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 10:51:06.494211 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":4,"thread_start_us":414,"threads_started":1}
W20260928 10:51:06.495155 10343 tablet.cc:2369] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:06.495815 10430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c725144aab1a4f6283d124b419fa42df" candidate_uuid: "cf989722d97442f7b8c79c199038a7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe9910c4d5b4304bafe35037caafa79" is_pre_election: true
I20260928 10:51:06.495803 10297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c725144aab1a4f6283d124b419fa42df" candidate_uuid: "cf989722d97442f7b8c79c199038a7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3447fe184f0547bb96a29fb3851c5b45" is_pre_election: true
I20260928 10:51:06.496548 10430 raft_consensus.cc:2454] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf989722d97442f7b8c79c199038a7ff in term 0.
I20260928 10:51:06.496567 10297 raft_consensus.cc:2454] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf989722d97442f7b8c79c199038a7ff in term 0.
I20260928 10:51:06.497475 10499 leader_election.cc:302] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe9910c4d5b4304bafe35037caafa79, cf989722d97442f7b8c79c199038a7ff; no voters:
I20260928 10:51:06.498023 10634 raft_consensus.cc:2781] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:06.498225 10634 raft_consensus.cc:482] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:06.498342 10634 raft_consensus.cc:3037] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:06.501183 10634 raft_consensus.cc:504] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.501911 10634 leader_election.cc:288] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [CANDIDATE]: Term 1 election: Requested vote from peers 3fe9910c4d5b4304bafe35037caafa79 (127.5.227.130:44379), 3447fe184f0547bb96a29fb3851c5b45 (127.5.227.129:43815)
I20260928 10:51:06.502856 10430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c725144aab1a4f6283d124b419fa42df" candidate_uuid: "cf989722d97442f7b8c79c199038a7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe9910c4d5b4304bafe35037caafa79"
I20260928 10:51:06.503090 10297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c725144aab1a4f6283d124b419fa42df" candidate_uuid: "cf989722d97442f7b8c79c199038a7ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3447fe184f0547bb96a29fb3851c5b45"
I20260928 10:51:06.503311 10297 raft_consensus.cc:3037] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:06.503315 10430 raft_consensus.cc:3037] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:06.508306 10430 raft_consensus.cc:2454] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf989722d97442f7b8c79c199038a7ff in term 1.
I20260928 10:51:06.508311 10297 raft_consensus.cc:2454] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf989722d97442f7b8c79c199038a7ff in term 1.
I20260928 10:51:06.509056 10497 leader_election.cc:302] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3447fe184f0547bb96a29fb3851c5b45, 3fe9910c4d5b4304bafe35037caafa79, cf989722d97442f7b8c79c199038a7ff; no voters:
I20260928 10:51:06.509521 10634 raft_consensus.cc:2781] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:06.510198 10634 raft_consensus.cc:686] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 1 LEADER]: Becoming Leader. State: Replica: cf989722d97442f7b8c79c199038a7ff, State: Running, Role: LEADER
I20260928 10:51:06.510951 10634 consensus_queue.cc:237] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } }
I20260928 10:51:06.517313 10152 catalog_manager.cc:5495] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff reported cstate change: term changed from 0 to 1, leader changed from <none> to cf989722d97442f7b8c79c199038a7ff (127.5.227.131). New cstate: current_term: 1 leader_uuid: "cf989722d97442f7b8c79c199038a7ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf989722d97442f7b8c79c199038a7ff" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 } health_report { overall_health: UNKNOWN } } }
W20260928 10:51:06.522707 10609 tablet.cc:2369] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:06.554663 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.547096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.5.227.131:39795)
0928 10:51:06.547369 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.547376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.547711 (+ 335us) server_negotiation.cc:407] Connection header received
0928 10:51:06.548236 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.548241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.548343 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.548526 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:06.549153 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.550004 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.551688 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.552366 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.552493 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.553139 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:06.553757 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:06.553882 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.554131 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.554216 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 10:51:06.566179 10297 raft_consensus.cc:1261] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 1 FOLLOWER]: Refusing update from remote peer cf989722d97442f7b8c79c199038a7ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:06.567046 10430 raft_consensus.cc:1261] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 1 FOLLOWER]: Refusing update from remote peer cf989722d97442f7b8c79c199038a7ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:06.567551 10634 consensus_queue.cc:1035] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3447fe184f0547bb96a29fb3851c5b45" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:06.568100 10637 consensus_queue.cc:1035] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe9910c4d5b4304bafe35037caafa79" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:06.583712 10640 mvcc.cc:205] Tried to move back new op lower bound from 7334267562237030400 to 7334267562035163136. Current Snapshot: MvccSnapshot[applied={T|T < 7334267562237030400}]
I20260928 10:51:06.586669 10641 mvcc.cc:205] Tried to move back new op lower bound from 7334267562237030400 to 7334267562035163136. Current Snapshot: MvccSnapshot[applied={T|T < 7334267562237030400}]
I20260928 10:51:06.606356 10152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:06.606917 10152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:06.611536 10152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f7372d0fcfe47e086901b1738d2da49: Sending DeleteTablet for 3 replicas of tablet c725144aab1a4f6283d124b419fa42df
I20260928 10:51:06.613622 10410 tablet_service.cc:1514] Processing DeleteTablet for tablet c725144aab1a4f6283d124b419fa42df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:06 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:42394
I20260928 10:51:06.613621 10543 tablet_service.cc:1514] Processing DeleteTablet for tablet c725144aab1a4f6283d124b419fa42df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:06 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:58070
I20260928 10:51:06.613672 10277 tablet_service.cc:1514] Processing DeleteTablet for tablet c725144aab1a4f6283d124b419fa42df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:06 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:56724
I20260928 10:51:06.618113 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 10215
I20260928 10:51:06.627668 10650 tablet_replica.cc:330] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: stopping tablet replica
I20260928 10:51:06.629278 10650 raft_consensus.cc:2229] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:06.631232 10650 raft_consensus.cc:2258] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:06.634578 10649 tablet_replica.cc:330] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: stopping tablet replica
I20260928 10:51:06.638180 10650 ts_tablet_manager.cc:1905] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:06.638908 10649 raft_consensus.cc:2229] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:06.639788 10649 raft_consensus.cc:2258] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:06.641300 10651 tablet_replica.cc:330] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: stopping tablet replica
I20260928 10:51:06.641841 10651 raft_consensus.cc:2229] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:06.642474 10651 raft_consensus.cc:2258] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:06.644178 10649 ts_tablet_manager.cc:1905] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:06.649780 10651 ts_tablet_manager.cc:1905] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 10:51:06 dist-test-slave-8w8g krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:51:06.653359 10650 ts_tablet_manager.cc:1918] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:06.664006 10650 log.cc:1196] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/c725144aab1a4f6283d124b419fa42df
I20260928 10:51:06.665380 10650 ts_tablet_manager.cc:1939] T c725144aab1a4f6283d124b419fa42df P 3fe9910c4d5b4304bafe35037caafa79: Deleting consensus metadata
I20260928 10:51:06.667331 10141 catalog_manager.cc:4841] TS 3fe9910c4d5b4304bafe35037caafa79 (127.5.227.130:44379): tablet c725144aab1a4f6283d124b419fa42df (table test-table [id=8f8e62f6374a43a5817e2d7229980cfa]) successfully deleted
I20260928 10:51:06.669265 10649 ts_tablet_manager.cc:1918] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:06.669785 10649 log.cc:1196] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/c725144aab1a4f6283d124b419fa42df
I20260928 10:51:06.670535 10649 ts_tablet_manager.cc:1939] T c725144aab1a4f6283d124b419fa42df P cf989722d97442f7b8c79c199038a7ff: Deleting consensus metadata
I20260928 10:51:06.670606 10651 ts_tablet_manager.cc:1918] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:06.670850 10651 log.cc:1196] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/c725144aab1a4f6283d124b419fa42df
I20260928 10:51:06.672062 10651 ts_tablet_manager.cc:1939] T c725144aab1a4f6283d124b419fa42df P 3447fe184f0547bb96a29fb3851c5b45: Deleting consensus metadata
I20260928 10:51:06.673573 10141 catalog_manager.cc:4841] TS cf989722d97442f7b8c79c199038a7ff (127.5.227.131:39795): tablet c725144aab1a4f6283d124b419fa42df (table test-table [id=8f8e62f6374a43a5817e2d7229980cfa]) successfully deleted
I20260928 10:51:06.674563 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.619598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.5.227.129:43815)
0928 10:51:06.619832 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.619842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.635926 (+ 16084us) server_negotiation.cc:407] Connection header received
0928 10:51:06.636029 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.636037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.636147 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.636275 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:06.638063 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.639206 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.642738 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.643412 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.643539 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.661737 (+ 18198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:06.661770 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:06.661795 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:06.661881 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:06.667298 (+ 5417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.668971 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.668984 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.668988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.669207 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.670001 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.670009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.670015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.670641 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:06.671168 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.672864 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.673344 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:51:06.675205 10139 catalog_manager.cc:4841] TS 3447fe184f0547bb96a29fb3851c5b45 (127.5.227.129:43815): tablet c725144aab1a4f6283d124b419fa42df (table test-table [id=8f8e62f6374a43a5817e2d7229980cfa]) successfully deleted
I20260928 10:51:06.676503 10337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:06.803442 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 10215
I20260928 10:51:06.828938 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 10348
Sep 28 10:51:06 dist-test-slave-8w8g krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:51:06.852715 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.829906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.5.227.130:44379)
0928 10:51:06.830155 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.830161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.830391 (+ 230us) server_negotiation.cc:407] Connection header received
0928 10:51:06.830567 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.830574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.830668 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.830804 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:06.832769 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.834668 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.837315 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.837797 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.838162 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.844427 (+ 6265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:06.844446 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:06.844463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:06.844511 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:06.848998 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.850136 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.850145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.850147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.850306 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:06.850826 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:06.850832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:06.850843 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:06.851315 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:06.851608 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:06.852079 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 10:51:06.852282 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 10:51:06.853811 10470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:06.959055 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 10348
I20260928 10:51:06.985529 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 10481
Sep 28 10:51:06 dist-test-slave-8w8g krb5kdc[10110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:51:07.008482 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:06.986587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.5.227.131:39795)
0928 10:51:06.986759 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:06.986766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:06.986897 (+ 131us) server_negotiation.cc:407] Connection header received
0928 10:51:06.987091 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:06.987101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:06.987197 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:06.987320 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:06.988706 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.989522 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.990854 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:06.991361 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:06.991525 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:06.998527 (+ 7002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:06.998540 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:06.998557 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:06.998611 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:07.002787 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:07.004312 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:07.004326 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:07.004329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:07.004700 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:07.005493 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:07.005499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:07.005502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:07.006112 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:07.006445 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:07.007879 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0928 10:51:07.008158 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:51:07.009426 10603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:07.147238 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 10481
I20260928 10:51:07.169842 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 10123
I20260928 10:51:07.171208 10184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:07.299466 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 10123
2026-09-28T10:51:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5135 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T10:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:07Z Disabled control of system clock
I20260928 10:51:07.386247 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40821
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 10:51:07.798076 10660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:07.798465 10660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:07.798631 10660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:07.806881 10660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:07.807041 10660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:07.807130 10660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:07.807242 10660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:07.807319 10660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:07.817451 10660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:40821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:07.818881 10660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:07.820855 10660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:07.833263 10668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:07.833324 10665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:07.833302 10666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:07.834617 10660 server_base.cc:1034] running on GCE node
I20260928 10:51:07.835572 10660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:07.837268 10660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:07.838546 10660 hybrid_clock.cc:648] HybridClock initialized: now 1790592667838505 us; error 45 us; skew 500 ppm
I20260928 10:51:07.838970 10660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:07.842296 10660 webserver.cc:460] Webserver started at http://127.5.227.190:39713/ using document root <none> and password file <none>
I20260928 10:51:07.843127 10660 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:07.843633 10660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:07.845250 10660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:07.849897 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "45500f98472f4498a19718893844f4bc"
format_stamp: "Formatted at 2026-09-28 10:51:07 on dist-test-slave-8w8g"
server_key: "614cf0b7f90dd1ff302db62b8c6b0fc7"
server_key_iv: "dd69cafd6e6c183d3d7b5896022d54d8"
server_key_version: "encryptionkey@0"
I20260928 10:51:07.851498 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "45500f98472f4498a19718893844f4bc"
format_stamp: "Formatted at 2026-09-28 10:51:07 on dist-test-slave-8w8g"
server_key: "614cf0b7f90dd1ff302db62b8c6b0fc7"
server_key_iv: "dd69cafd6e6c183d3d7b5896022d54d8"
server_key_version: "encryptionkey@0"
I20260928 10:51:07.859304 10660 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260928 10:51:07.863948 10674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:07.866736 10660 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 10:51:07.867156 10660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "45500f98472f4498a19718893844f4bc"
format_stamp: "Formatted at 2026-09-28 10:51:07 on dist-test-slave-8w8g"
server_key: "614cf0b7f90dd1ff302db62b8c6b0fc7"
server_key_iv: "dd69cafd6e6c183d3d7b5896022d54d8"
server_key_version: "encryptionkey@0"
I20260928 10:51:07.867461 10660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:07.875171 10660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:07.876375 10660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:07.876741 10660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:07.901973 10660 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:40821
I20260928 10:51:07.901980 10725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:40821 every 8 connection(s)
I20260928 10:51:07.904143 10660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:07.908928 10726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:07.913249 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 10660
I20260928 10:51:07.913496 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:07.914362 6030 external_mini_cluster.cc:1428] Setting key 4b66da9dd327fbd51a079c01a64125ed
I20260928 10:51:07.923100 10726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc: Bootstrap starting.
I20260928 10:51:07.926818 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:07.917653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.5.227.190:40821)
0928 10:51:07.919156 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:07.919171 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:07.919237 (+ 66us) server_negotiation.cc:407] Connection header received
0928 10:51:07.921433 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:07.921465 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:07.922717 (+ 1252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:07.923512 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:07.924805 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:07.924811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:07.924825 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:07.924860 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:07.924928 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:07.924952 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:07.925215 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:07.925394 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 10:51:07.925718 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":431,"threads_started":1}
I20260928 10:51:07.927259 10726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:07.928903 10726 log.cc:824] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:07.933013 10726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc: No bootstrap required, opened a new log
I20260928 10:51:07.941596 10726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45500f98472f4498a19718893844f4bc" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40821 } }
I20260928 10:51:07.942068 10726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:07.942185 10726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45500f98472f4498a19718893844f4bc, State: Initialized, Role: FOLLOWER
I20260928 10:51:07.942898 10726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45500f98472f4498a19718893844f4bc" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40821 } }
I20260928 10:51:07.943186 10726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:07.943346 10726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:07.943540 10726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:07.946835 10726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45500f98472f4498a19718893844f4bc" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40821 } }
I20260928 10:51:07.947489 10726 leader_election.cc:302] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45500f98472f4498a19718893844f4bc; no voters:
I20260928 10:51:07.948341 10726 leader_election.cc:288] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:07.948520 10731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:07.950070 10731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [term 1 LEADER]: Becoming Leader. State: Replica: 45500f98472f4498a19718893844f4bc, State: Running, Role: LEADER
I20260928 10:51:07.950692 10731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45500f98472f4498a19718893844f4bc" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40821 } }
I20260928 10:51:07.951624 10726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:07.954053 10732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45500f98472f4498a19718893844f4bc. Latest consensus state: current_term: 1 leader_uuid: "45500f98472f4498a19718893844f4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45500f98472f4498a19718893844f4bc" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40821 } } }
I20260928 10:51:07.954500 10732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:07.955365 10733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45500f98472f4498a19718893844f4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45500f98472f4498a19718893844f4bc" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40821 } } }
I20260928 10:51:07.956753 10733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:07.959378 10736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:07.965648 10736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:07.976671 10736 catalog_manager.cc:1347] Generated new cluster ID: c0fcb016d3744feea36b26e07f75960e
I20260928 10:51:07.976816 10736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:07.999377 10736 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:07.999992 10736 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:08.012708 10736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc: Generated new TSK 0
I20260928 10:51:08.013742 10736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:51:08.020056 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40821
--builtin_ntp_servers=127.5.227.148:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 10:51:08.449256 10750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:08.449656 10750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:08.449841 10750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:08.460544 10750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:08.460721 10750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:08.460908 10750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:08.471968 10750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:08.473956 10750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:08.476681 10750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:08.495468 10756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:08.495494 10755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:08.495494 10758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:08.497042 10750 server_base.cc:1034] running on GCE node
I20260928 10:51:08.497828 10750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:08.499058 10750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:08.500352 10750 hybrid_clock.cc:648] HybridClock initialized: now 1790592668500266 us; error 74 us; skew 500 ppm
I20260928 10:51:08.500845 10750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:08.504135 10750 webserver.cc:460] Webserver started at http://127.5.227.129:39503/ using document root <none> and password file <none>
I20260928 10:51:08.505016 10750 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:08.505142 10750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:08.505626 10750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:08.510340 10750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "2d1ec50279484168ada5e0b4766244ef"
format_stamp: "Formatted at 2026-09-28 10:51:08 on dist-test-slave-8w8g"
server_key: "360280b8f8e5506be65baaff9ceec01f"
server_key_iv: "1f7eb68682803c96cea81a1c732cec4c"
server_key_version: "encryptionkey@0"
I20260928 10:51:08.511650 10750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "2d1ec50279484168ada5e0b4766244ef"
format_stamp: "Formatted at 2026-09-28 10:51:08 on dist-test-slave-8w8g"
server_key: "360280b8f8e5506be65baaff9ceec01f"
server_key_iv: "1f7eb68682803c96cea81a1c732cec4c"
server_key_version: "encryptionkey@0"
I20260928 10:51:08.518656 10750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:51:08.523355 10764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:08.526093 10750 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 10:51:08.526434 10750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "2d1ec50279484168ada5e0b4766244ef"
format_stamp: "Formatted at 2026-09-28 10:51:08 on dist-test-slave-8w8g"
server_key: "360280b8f8e5506be65baaff9ceec01f"
server_key_iv: "1f7eb68682803c96cea81a1c732cec4c"
server_key_version: "encryptionkey@0"
I20260928 10:51:08.526764 10750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:08.537060 10750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:08.538424 10750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:08.538833 10750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:08.540130 10750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:08.542232 10750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:08.542446 10750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:08.542765 10750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:08.543146 10750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:08.588596 10750 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:34575
I20260928 10:51:08.588676 10876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:34575 every 8 connection(s)
I20260928 10:51:08.590458 10750 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:08.600159 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 10750
I20260928 10:51:08.600423 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:51:08.601079 6030 external_mini_cluster.cc:1428] Setting key 1c28aa92d2cf7a41cc7180d5b6c4ea35
I20260928 10:51:08.603556 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:08.594254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:36099 (local address 127.5.227.190:40821)
0928 10:51:08.594754 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:08.594759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:08.596786 (+ 2027us) server_negotiation.cc:407] Connection header received
0928 10:51:08.599570 (+ 2784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:08.599576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:08.599659 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:08.599916 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:08.602444 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:08.602450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:08.602454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:08.602477 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:08.602530 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:08.602551 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:08.602691 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:08.603075 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 10:51:08.603151 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":265,"threads_started":1}
I20260928 10:51:08.605165 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:08.595239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40821 (local address 127.5.227.129:36099)
0928 10:51:08.596580 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:08.596627 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:08.599094 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:08.600385 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:08.600407 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:08.600581 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:08.601866 (+ 1285us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:08.601869 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:08.601965 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:08.602111 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:08.602699 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:08.602709 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:08.602717 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:51:08.603281 (+ 564us) client_negotiation.cc:240] Negotiation successful
0928 10:51:08.603608 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":480,"threads_started":1}
I20260928 10:51:08.605350 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40821
--builtin_ntp_servers=127.5.227.148:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 10:51:08.607921 10877 heartbeater.cc:347] Connected to a master server at 127.5.227.190:40821
I20260928 10:51:08.608415 10877 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:08.609488 10877 heartbeater.cc:511] Master 127.5.227.190:40821 requested a full tablet report, sending...
I20260928 10:51:08.613023 10691 ts_manager.cc:194] Registered new tserver with Master: 2d1ec50279484168ada5e0b4766244ef (127.5.227.129:34575)
W20260928 10:51:09.037966 10881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:09.038300 10881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:09.038446 10881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:09.046319 10881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:09.046451 10881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:09.046568 10881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:09.056656 10881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:09.058185 10881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:09.060361 10881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:09.077224 10887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:09.077243 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:09.077368 10886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:09.078199 10881 server_base.cc:1034] running on GCE node
I20260928 10:51:09.079084 10881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:09.080367 10881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:09.081702 10881 hybrid_clock.cc:648] HybridClock initialized: now 1790592669081615 us; error 66 us; skew 500 ppm
I20260928 10:51:09.082360 10881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:09.085770 10881 webserver.cc:460] Webserver started at http://127.5.227.130:37693/ using document root <none> and password file <none>
I20260928 10:51:09.086769 10881 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:09.086937 10881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:09.087322 10881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:09.090866 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "72a3e0d6e2174db8b806b2bf0cd5733a"
format_stamp: "Formatted at 2026-09-28 10:51:09 on dist-test-slave-8w8g"
server_key: "4eb2411efb0e162630ecdfa5daaafd0c"
server_key_iv: "90d5c19a1c0ca9e59c4926565901917d"
server_key_version: "encryptionkey@0"
I20260928 10:51:09.091881 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "72a3e0d6e2174db8b806b2bf0cd5733a"
format_stamp: "Formatted at 2026-09-28 10:51:09 on dist-test-slave-8w8g"
server_key: "4eb2411efb0e162630ecdfa5daaafd0c"
server_key_iv: "90d5c19a1c0ca9e59c4926565901917d"
server_key_version: "encryptionkey@0"
I20260928 10:51:09.098744 10881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:51:09.104494 10895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:09.106607 10881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 10:51:09.106877 10881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "72a3e0d6e2174db8b806b2bf0cd5733a"
format_stamp: "Formatted at 2026-09-28 10:51:09 on dist-test-slave-8w8g"
server_key: "4eb2411efb0e162630ecdfa5daaafd0c"
server_key_iv: "90d5c19a1c0ca9e59c4926565901917d"
server_key_version: "encryptionkey@0"
I20260928 10:51:09.107193 10881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:09.114754 10881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:09.115932 10881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:09.116210 10881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:09.117254 10881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:09.118885 10881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:09.118985 10881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:09.119074 10881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:09.119130 10881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:09.167164 10881 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:37309
I20260928 10:51:09.167537 11007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:37309 every 8 connection(s)
I20260928 10:51:09.169220 10881 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:09.176894 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 10881
I20260928 10:51:09.177096 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:51:09.177664 6030 external_mini_cluster.cc:1428] Setting key 64986b34d1243c0c1ac6f58ff080d726
I20260928 10:51:09.181667 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40821
--builtin_ntp_servers=127.5.227.148:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 10:51:09.181957 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.173000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:47153 (local address 127.5.227.190:40821)
0928 10:51:09.173525 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.173530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.175372 (+ 1842us) server_negotiation.cc:407] Connection header received
0928 10:51:09.178197 (+ 2825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.178205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.178279 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.178432 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.180909 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.180913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.180917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.180935 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.180982 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.180994 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.181108 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.181479 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.181621 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":290,"threads_started":1}
I20260928 10:51:09.183403 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.173715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40821 (local address 127.5.227.130:47153)
0928 10:51:09.175165 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:09.175208 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:09.177623 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:09.179008 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:09.179023 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:09.179256 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:09.180172 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:09.180174 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:09.180223 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:09.180553 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:09.181119 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:09.181131 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:09.181140 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 10:51:09.181488 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 10:51:09.181854 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":421,"threads_started":1}
I20260928 10:51:09.185565 11008 heartbeater.cc:347] Connected to a master server at 127.5.227.190:40821
I20260928 10:51:09.186157 11008 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:09.187220 11008 heartbeater.cc:511] Master 127.5.227.190:40821 requested a full tablet report, sending...
I20260928 10:51:09.189647 10691 ts_manager.cc:194] Registered new tserver with Master: 72a3e0d6e2174db8b806b2bf0cd5733a (127.5.227.130:37309)
W20260928 10:51:09.606279 11012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:09.606616 11012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:09.606714 11012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:09.616590 11012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:09.616784 11012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:09.616909 11012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:09.617883 10877 heartbeater.cc:503] Master 127.5.227.190:40821 was elected leader, sending a full tablet report...
I20260928 10:51:09.627599 11012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:09.629063 11012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:09.631013 11012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:09.652351 11017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:09.653013 11020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:09.653052 11018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:09.654317 11012 server_base.cc:1034] running on GCE node
I20260928 10:51:09.655434 11012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:09.657490 11012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:09.661149 11012 hybrid_clock.cc:648] HybridClock initialized: now 1790592669661129 us; error 82 us; skew 500 ppm
I20260928 10:51:09.661907 11012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:09.665891 11012 webserver.cc:460] Webserver started at http://127.5.227.131:43947/ using document root <none> and password file <none>
I20260928 10:51:09.666695 11012 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:09.666806 11012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:09.667109 11012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:09.670439 11012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21"
format_stamp: "Formatted at 2026-09-28 10:51:09 on dist-test-slave-8w8g"
server_key: "43ef752ae82185dfc90cf48b332342ce"
server_key_iv: "768d50f683ff6a94d5f33c5be83b1504"
server_key_version: "encryptionkey@0"
I20260928 10:51:09.671705 11012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21"
format_stamp: "Formatted at 2026-09-28 10:51:09 on dist-test-slave-8w8g"
server_key: "43ef752ae82185dfc90cf48b332342ce"
server_key_iv: "768d50f683ff6a94d5f33c5be83b1504"
server_key_version: "encryptionkey@0"
I20260928 10:51:09.680665 11012 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260928 10:51:09.687992 11026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:09.690174 11012 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260928 10:51:09.690452 11012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21"
format_stamp: "Formatted at 2026-09-28 10:51:09 on dist-test-slave-8w8g"
server_key: "43ef752ae82185dfc90cf48b332342ce"
server_key_iv: "768d50f683ff6a94d5f33c5be83b1504"
server_key_version: "encryptionkey@0"
I20260928 10:51:09.690760 11012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:09.698792 11012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:09.700434 11012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:09.701090 11012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:09.702762 11012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:09.705868 11012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:09.706044 11012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:09.706207 11012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:09.706321 11012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:09.754312 11012 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:37169
I20260928 10:51:09.754346 11138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:37169 every 8 connection(s)
I20260928 10:51:09.756728 11012 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:09.762045 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11012
I20260928 10:51:09.762259 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:51:09.763347 6030 external_mini_cluster.cc:1428] Setting key 69c55f00c20baff5e326dea1190968e4
I20260928 10:51:09.771409 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.762048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:44561 (local address 127.5.227.190:40821)
0928 10:51:09.765087 (+ 3039us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.765094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.765119 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:51:09.767143 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.767149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.767333 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.767476 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.769810 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.769816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.769819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.769841 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.769889 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.769903 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.770440 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.771032 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.771113 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2604,"thread_start_us":2479,"threads_started":1}
I20260928 10:51:09.772926 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.762698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40821 (local address 127.5.227.131:44561)
0928 10:51:09.764386 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:09.764449 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:09.766500 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:09.767951 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:09.767962 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:09.768085 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:09.769109 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:09.769112 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:09.769162 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:09.769408 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:09.770554 (+ 1146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:09.770564 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:09.770572 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:51:09.770974 (+ 402us) client_negotiation.cc:240] Negotiation successful
0928 10:51:09.771253 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":427,"threads_started":1}
I20260928 10:51:09.774998 11139 heartbeater.cc:347] Connected to a master server at 127.5.227.190:40821
I20260928 10:51:09.775413 11139 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:09.776187 11139 heartbeater.cc:511] Master 127.5.227.190:40821 requested a full tablet report, sending...
I20260928 10:51:09.778384 10691 ts_manager.cc:194] Registered new tserver with Master: 5ac2ebd93aa74fccb36c92fdca2c7e21 (127.5.227.131:37169)
I20260928 10:51:09.782295 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:51:09.793445 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.789611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.5.227.190:40821)
0928 10:51:09.789952 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.789960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.790713 (+ 753us) server_negotiation.cc:407] Connection header received
0928 10:51:09.790944 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.790956 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.791013 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.791131 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.792461 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.792466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.792469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.792492 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.792529 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.792542 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.792678 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.792927 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.792987 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260928 10:51:09.801016 10691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:51:09.810595 10691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:09.839411 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.828675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:34575 (local address 127.0.0.1:35378)
0928 10:51:09.830211 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:09.830278 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:09.830460 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:09.833227 (+ 2767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:09.833239 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:09.833355 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:09.834572 (+ 1217us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:09.834577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:09.834621 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:09.834748 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:09.838136 (+ 3388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:09.838162 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:09.838179 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 10:51:09.838457 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 10:51:09.838617 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":1373,"threads_started":1}
I20260928 10:51:09.841985 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.829071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.5.227.129:34575)
0928 10:51:09.832003 (+ 2932us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.832017 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.832062 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:51:09.832164 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.832178 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.832662 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.834086 (+ 1424us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.837530 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.837546 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.837569 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.837599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.837703 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.837738 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.840658 (+ 2920us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.840780 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.841300 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2633,"thread_start_us":1993,"threads_started":1}
I20260928 10:51:09.848512 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.829820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.5.227.130:37309)
0928 10:51:09.843123 (+ 13303us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.843136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.843181 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:51:09.843306 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.843319 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.844088 (+ 769us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.845322 (+ 1234us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.847008 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.847026 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.847050 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.847105 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.847194 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.847342 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.847683 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.847828 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.848109 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13043,"thread_start_us":14124,"threads_started":1}
I20260928 10:51:09.848259 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.829374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:37309 (local address 127.0.0.1:39560)
0928 10:51:09.839851 (+ 10477us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:09.839880 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:09.840027 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:09.844612 (+ 4585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:09.844631 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:09.844710 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:09.845711 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:09.845714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:09.845733 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:09.845856 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:09.847616 (+ 1760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:09.847621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:09.847639 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 10:51:09.847798 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 10:51:09.847888 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10266,"thread_start_us":15329,"threads_started":1}
I20260928 10:51:09.849888 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.831827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.5.227.131:37169)
0928 10:51:09.841769 (+ 9942us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.841889 (+ 120us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.845410 (+ 3521us) server_negotiation.cc:407] Connection header received
0928 10:51:09.846193 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.846204 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.846572 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.846738 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.848976 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.848986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.849003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.849028 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.849105 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.849135 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.849333 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.849439 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.849552 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9664,"thread_start_us":9480,"threads_started":1}
I20260928 10:51:09.850214 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.831625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:37169 (local address 127.0.0.1:44742)
0928 10:51:09.845212 (+ 13587us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:09.845243 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:09.846015 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:09.847731 (+ 1716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:09.847736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:09.847788 (+ 52us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:09.848710 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:09.848714 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:09.848732 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:09.848782 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:09.849291 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:09.849296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:09.849302 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:51:09.849463 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 10:51:09.849538 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13382,"thread_start_us":16663,"threads_started":1}
I20260928 10:51:09.853978 10943 tablet_service.cc:1467] Processing CreateTablet for tablet cef9aedfc8414174ac5fb005ac3bd172 (DEFAULT_TABLE table=test-table [id=4b8ad7f9b7d34a63bf5ee886ccc59430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:09.853978 10812 tablet_service.cc:1467] Processing CreateTablet for tablet cef9aedfc8414174ac5fb005ac3bd172 (DEFAULT_TABLE table=test-table [id=4b8ad7f9b7d34a63bf5ee886ccc59430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:09.855870 10943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef9aedfc8414174ac5fb005ac3bd172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:09.855911 10812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef9aedfc8414174ac5fb005ac3bd172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:09.857610 11074 tablet_service.cc:1467] Processing CreateTablet for tablet cef9aedfc8414174ac5fb005ac3bd172 (DEFAULT_TABLE table=test-table [id=4b8ad7f9b7d34a63bf5ee886ccc59430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:09.860617 11074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef9aedfc8414174ac5fb005ac3bd172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:09.878965 11158 tablet_bootstrap.cc:492] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Bootstrap starting.
I20260928 10:51:09.881023 11159 tablet_bootstrap.cc:492] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Bootstrap starting.
I20260928 10:51:09.883481 11160 tablet_bootstrap.cc:492] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Bootstrap starting.
I20260928 10:51:09.885911 11158 tablet_bootstrap.cc:654] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:09.887614 11159 tablet_bootstrap.cc:654] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:09.888136 11160 tablet_bootstrap.cc:654] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:09.888147 11158 log.cc:824] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:09.889600 11159 log.cc:824] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:09.889866 11160 log.cc:824] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:09.893537 11160 tablet_bootstrap.cc:492] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: No bootstrap required, opened a new log
I20260928 10:51:09.893996 11158 tablet_bootstrap.cc:492] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: No bootstrap required, opened a new log
I20260928 10:51:09.893996 11159 tablet_bootstrap.cc:492] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: No bootstrap required, opened a new log
I20260928 10:51:09.894326 11159 ts_tablet_manager.cc:1397] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260928 10:51:09.894326 11158 ts_tablet_manager.cc:1397] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20260928 10:51:09.894347 11160 ts_tablet_manager.cc:1397] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 10:51:09.908252 11160 raft_consensus.cc:348] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.908861 11160 raft_consensus.cc:374] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:09.908457 11158 raft_consensus.cc:348] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.908459 11159 raft_consensus.cc:348] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.908965 11160 raft_consensus.cc:729] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a3e0d6e2174db8b806b2bf0cd5733a, State: Initialized, Role: FOLLOWER
I20260928 10:51:09.909037 11158 raft_consensus.cc:374] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:09.909036 11159 raft_consensus.cc:374] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:09.909161 11158 raft_consensus.cc:729] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac2ebd93aa74fccb36c92fdca2c7e21, State: Initialized, Role: FOLLOWER
I20260928 10:51:09.909160 11159 raft_consensus.cc:729] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1ec50279484168ada5e0b4766244ef, State: Initialized, Role: FOLLOWER
I20260928 10:51:09.909891 11159 consensus_queue.cc:260] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.909998 11160 consensus_queue.cc:260] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.910091 11158 consensus_queue.cc:260] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.914351 11160 ts_tablet_manager.cc:1428] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260928 10:51:09.915313 11008 heartbeater.cc:503] Master 127.5.227.190:40821 was elected leader, sending a full tablet report...
W20260928 10:51:09.922639 11009 tablet.cc:2369] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:09.928680 11139 heartbeater.cc:503] Master 127.5.227.190:40821 was elected leader, sending a full tablet report...
I20260928 10:51:09.929523 11159 ts_tablet_manager.cc:1428] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Time spent starting tablet: real 0.035s user 0.017s sys 0.017s
I20260928 10:51:09.931391 11158 ts_tablet_manager.cc:1428] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Time spent starting tablet: real 0.037s user 0.009s sys 0.027s
I20260928 10:51:09.975579 11165 raft_consensus.cc:482] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:09.976074 11165 raft_consensus.cc:504] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.978837 11165 leader_election.cc:288] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d1ec50279484168ada5e0b4766244ef (127.5.227.129:34575), 72a3e0d6e2174db8b806b2bf0cd5733a (127.5.227.130:37309)
I20260928 10:51:09.989965 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.979310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:34575 (local address 127.5.227.131:52291)
0928 10:51:09.979612 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:09.979634 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:09.979931 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:09.980520 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:09.980525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:09.980552 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:09.988554 (+ 8002us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:09.988559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:09.988586 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:09.988666 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:09.989325 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:09.989332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:09.989339 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:51:09.989525 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 10:51:09.989617 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 10:51:09.989996 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.979334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:52291 (local address 127.5.227.129:34575)
0928 10:51:09.979721 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.979752 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.979779 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:51:09.980146 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.980153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.980227 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.980399 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.989046 (+ 8647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.989053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.989058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.989086 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.989134 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.989152 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.989308 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.989634 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.989709 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260928 10:51:09.990067 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.979673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:55243 (local address 127.5.227.130:37309)
0928 10:51:09.979925 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:09.979934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:09.980551 (+ 617us) server_negotiation.cc:407] Connection header received
0928 10:51:09.980733 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:09.980738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:09.980797 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:09.980987 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:09.989050 (+ 8063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:09.989057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:09.989061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:09.989088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:09.989134 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:09.989152 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:09.989307 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:09.989632 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 10:51:09.989709 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:51:09.990201 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:09.979446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:37309 (local address 127.5.227.131:55243)
0928 10:51:09.980421 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:09.980465 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:09.980579 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:09.981257 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:09.981264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:09.981289 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:09.988554 (+ 7265us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:51:09.988559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:09.988586 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:51:09.988666 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:09.989325 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:09.989332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:09.989339 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:51:09.989528 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 10:51:09.989657 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":153,"thread_start_us":537,"threads_started":1}
I20260928 10:51:09.991284 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef9aedfc8414174ac5fb005ac3bd172" candidate_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d1ec50279484168ada5e0b4766244ef" is_pre_election: true
I20260928 10:51:09.991281 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef9aedfc8414174ac5fb005ac3bd172" candidate_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" is_pre_election: true
I20260928 10:51:09.991926 10832 raft_consensus.cc:2454] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ac2ebd93aa74fccb36c92fdca2c7e21 in term 0.
I20260928 10:51:09.991930 10963 raft_consensus.cc:2454] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ac2ebd93aa74fccb36c92fdca2c7e21 in term 0.
I20260928 10:51:09.992699 11027 leader_election.cc:302] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac2ebd93aa74fccb36c92fdca2c7e21, 72a3e0d6e2174db8b806b2bf0cd5733a; no voters:
I20260928 10:51:09.993168 11165 raft_consensus.cc:2781] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:09.993330 11165 raft_consensus.cc:482] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:09.993450 11165 raft_consensus.cc:3037] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:09.996592 11165 raft_consensus.cc:504] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
I20260928 10:51:09.997697 11165 leader_election.cc:288] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [CANDIDATE]: Term 1 election: Requested vote from peers 2d1ec50279484168ada5e0b4766244ef (127.5.227.129:34575), 72a3e0d6e2174db8b806b2bf0cd5733a (127.5.227.130:37309)
I20260928 10:51:09.998471 10832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef9aedfc8414174ac5fb005ac3bd172" candidate_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d1ec50279484168ada5e0b4766244ef"
I20260928 10:51:09.998729 10832 raft_consensus.cc:3037] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:09.998616 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef9aedfc8414174ac5fb005ac3bd172" candidate_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a"
I20260928 10:51:09.999099 10963 raft_consensus.cc:3037] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:10.001927 10832 raft_consensus.cc:2454] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ac2ebd93aa74fccb36c92fdca2c7e21 in term 1.
I20260928 10:51:10.003608 11028 leader_election.cc:302] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d1ec50279484168ada5e0b4766244ef, 5ac2ebd93aa74fccb36c92fdca2c7e21; no voters:
I20260928 10:51:10.004034 10963 raft_consensus.cc:2454] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ac2ebd93aa74fccb36c92fdca2c7e21 in term 1.
I20260928 10:51:10.004251 11165 raft_consensus.cc:2781] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:10.005496 11165 raft_consensus.cc:686] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 1 LEADER]: Becoming Leader. State: Replica: 5ac2ebd93aa74fccb36c92fdca2c7e21, State: Running, Role: LEADER
I20260928 10:51:10.006386 11165 consensus_queue.cc:237] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } }
W20260928 10:51:10.009573 11140 tablet.cc:2369] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:10.013320 10691 catalog_manager.cc:5495] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ac2ebd93aa74fccb36c92fdca2c7e21 (127.5.227.131). New cstate: current_term: 1 leader_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ac2ebd93aa74fccb36c92fdca2c7e21" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 } health_report { overall_health: UNKNOWN } } }
I20260928 10:51:10.070669 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:10.061797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.5.227.131:37169)
0928 10:51:10.062047 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:10.062055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:10.062084 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:51:10.062233 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:10.062237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:10.062423 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:10.062569 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:10.069943 (+ 7374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:10.069948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:10.069952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:10.069975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:10.070021 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:10.070037 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:10.070203 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:10.070322 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 10:51:10.070390 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 10:51:10.082260 10832 raft_consensus.cc:1261] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 1 FOLLOWER]: Refusing update from remote peer 5ac2ebd93aa74fccb36c92fdca2c7e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:10.082566 10963 raft_consensus.cc:1261] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 1 FOLLOWER]: Refusing update from remote peer 5ac2ebd93aa74fccb36c92fdca2c7e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:10.083784 11168 consensus_queue.cc:1035] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d1ec50279484168ada5e0b4766244ef" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:10.084355 11165 consensus_queue.cc:1035] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a3e0d6e2174db8b806b2bf0cd5733a" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:51:10.094484 10878 tablet.cc:2369] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:10.124275 10691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:10.124682 10691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:10.129518 10691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45500f98472f4498a19718893844f4bc: Sending DeleteTablet for 3 replicas of tablet cef9aedfc8414174ac5fb005ac3bd172
I20260928 10:51:10.131505 10812 tablet_service.cc:1514] Processing DeleteTablet for tablet cef9aedfc8414174ac5fb005ac3bd172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:10 UTC) from {username='slave'} at 127.0.0.1:35378
I20260928 10:51:10.131635 11074 tablet_service.cc:1514] Processing DeleteTablet for tablet cef9aedfc8414174ac5fb005ac3bd172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:10 UTC) from {username='slave'} at 127.0.0.1:44742
I20260928 10:51:10.132489 10943 tablet_service.cc:1514] Processing DeleteTablet for tablet cef9aedfc8414174ac5fb005ac3bd172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:10 UTC) from {username='slave'} at 127.0.0.1:39560
I20260928 10:51:10.133576 11181 tablet_replica.cc:330] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: stopping tablet replica
I20260928 10:51:10.134235 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 10750
I20260928 10:51:10.134308 11181 raft_consensus.cc:2229] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:10.135547 11181 raft_consensus.cc:2258] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:10.142297 11181 ts_tablet_manager.cc:1905] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:10.147594 11182 tablet_replica.cc:330] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: stopping tablet replica
I20260928 10:51:10.148236 11182 raft_consensus.cc:2229] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:10.149235 11182 raft_consensus.cc:2258] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:10.151813 11182 ts_tablet_manager.cc:1905] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:10.152698 11181 ts_tablet_manager.cc:1918] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:10.152984 11181 log.cc:1196] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/cef9aedfc8414174ac5fb005ac3bd172
I20260928 10:51:10.154036 11181 ts_tablet_manager.cc:1939] T cef9aedfc8414174ac5fb005ac3bd172 P 2d1ec50279484168ada5e0b4766244ef: Deleting consensus metadata
I20260928 10:51:10.157045 10676 catalog_manager.cc:4841] TS 2d1ec50279484168ada5e0b4766244ef (127.5.227.129:34575): tablet cef9aedfc8414174ac5fb005ac3bd172 (table test-table [id=4b8ad7f9b7d34a63bf5ee886ccc59430]) successfully deleted
I20260928 10:51:10.159618 11183 tablet_replica.cc:330] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: stopping tablet replica
I20260928 10:51:10.160439 11183 raft_consensus.cc:2229] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:10.161304 11183 raft_consensus.cc:2258] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:10.161789 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:10.136668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.5.227.129:34575)
0928 10:51:10.137044 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:10.137052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:10.158421 (+ 21369us) server_negotiation.cc:407] Connection header received
0928 10:51:10.159001 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:10.159025 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:10.159163 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:10.159280 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:10.160902 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:10.160910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:10.160916 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:10.160938 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:10.160983 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:10.160999 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:10.161113 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:10.161393 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 10:51:10.161466 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:51:10.163301 10872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:10.163414 11182 ts_tablet_manager.cc:1918] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:10.163647 11182 log.cc:1196] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/cef9aedfc8414174ac5fb005ac3bd172
I20260928 10:51:10.164950 11182 ts_tablet_manager.cc:1939] T cef9aedfc8414174ac5fb005ac3bd172 P 5ac2ebd93aa74fccb36c92fdca2c7e21: Deleting consensus metadata
I20260928 10:51:10.166282 11183 ts_tablet_manager.cc:1905] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:10.175444 11183 ts_tablet_manager.cc:1918] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:10.176170 11183 log.cc:1196] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/cef9aedfc8414174ac5fb005ac3bd172
I20260928 10:51:10.177368 11183 ts_tablet_manager.cc:1939] T cef9aedfc8414174ac5fb005ac3bd172 P 72a3e0d6e2174db8b806b2bf0cd5733a: Deleting consensus metadata
I20260928 10:51:10.179616 10676 catalog_manager.cc:4841] TS 5ac2ebd93aa74fccb36c92fdca2c7e21 (127.5.227.131:37169): tablet cef9aedfc8414174ac5fb005ac3bd172 (table test-table [id=4b8ad7f9b7d34a63bf5ee886ccc59430]) successfully deleted
I20260928 10:51:10.182184 10675 catalog_manager.cc:4841] TS 72a3e0d6e2174db8b806b2bf0cd5733a (127.5.227.130:37309): tablet cef9aedfc8414174ac5fb005ac3bd172 (table test-table [id=4b8ad7f9b7d34a63bf5ee886ccc59430]) successfully deleted
I20260928 10:51:10.346499 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 10750
I20260928 10:51:10.367535 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 10881
I20260928 10:51:10.371825 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:10.368653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.5.227.130:37309)
0928 10:51:10.368930 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:10.368937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:10.369293 (+ 356us) server_negotiation.cc:407] Connection header received
0928 10:51:10.369519 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:10.369527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:10.369658 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:10.369790 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:10.370966 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:10.370973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:10.370979 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:10.371003 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:10.371046 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:10.371063 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:10.371220 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:10.371333 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 10:51:10.371401 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:51:10.372684 11003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:10.501302 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 10881
I20260928 10:51:10.521152 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11012
I20260928 10:51:10.525208 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:10.522562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.5.227.131:37169)
0928 10:51:10.522795 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:10.522803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:10.522828 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:51:10.522947 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:10.522953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:10.523027 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:10.523161 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:10.524394 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:10.524401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:10.524405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:10.524426 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:10.524474 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:10.524509 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:10.524671 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:10.524818 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 10:51:10.524885 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:51:10.526147 11134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:10.669988 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11012
I20260928 10:51:10.692256 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 10660
I20260928 10:51:10.693820 10721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:10.811443 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 10660
2026-09-28T10:51:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3508 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 10:51:10.856622 6030 mini_oidc.cc:152] Starting JWKS server
I20260928 10:51:10.859639 6030 webserver.cc:460] Webserver started at https://127.0.0.1:46697/ using document root <none> and password file <none>
I20260928 10:51:10.860221 6030 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:51:10.861079 6030 webserver.cc:460] Webserver started at http://127.0.0.1:37083/ using document root <none> and password file <none>
2026-09-28T10:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:10Z Disabled control of system clock
I20260928 10:51:10.893958 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40127
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:40127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/testchainca.cert with env {}
W20260928 10:51:11.284384 11194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:11.284761 11194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:11.285049 11194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:11.294051 11194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:11.294227 11194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:11.294272 11194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:11.294322 11194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:11.294360 11194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:51:11.294481 11194 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:51:11.294552 11194 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46697/jwks.json
I20260928 10:51:11.306977 11194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:40127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:11.308614 11194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:11.310195 11194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:11.326675 11200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:11.326591 11202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:11.326552 11199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:11.327690 11194 server_base.cc:1034] running on GCE node
I20260928 10:51:11.328791 11194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:11.330564 11194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:11.332095 11194 hybrid_clock.cc:648] HybridClock initialized: now 1790592671332039 us; error 51 us; skew 500 ppm
I20260928 10:51:11.332829 11194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:11.336395 11194 webserver.cc:460] Webserver started at http://127.5.227.190:41833/ using document root <none> and password file <none>
I20260928 10:51:11.337646 11194 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:11.337981 11194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:11.338562 11194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:11.343299 11194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "05018657ae284e6f86da505e5f4b6321"
format_stamp: "Formatted at 2026-09-28 10:51:11 on dist-test-slave-8w8g"
server_key: "59e59ed13b782f197f2212575a083fc9"
server_key_iv: "ec6e856b6d1bc11802482be11942e029"
server_key_version: "encryptionkey@0"
I20260928 10:51:11.344328 11194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "05018657ae284e6f86da505e5f4b6321"
format_stamp: "Formatted at 2026-09-28 10:51:11 on dist-test-slave-8w8g"
server_key: "59e59ed13b782f197f2212575a083fc9"
server_key_iv: "ec6e856b6d1bc11802482be11942e029"
server_key_version: "encryptionkey@0"
I20260928 10:51:11.351465 11194 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260928 10:51:11.356671 11208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:11.358829 11194 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260928 10:51:11.359074 11194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "05018657ae284e6f86da505e5f4b6321"
format_stamp: "Formatted at 2026-09-28 10:51:11 on dist-test-slave-8w8g"
server_key: "59e59ed13b782f197f2212575a083fc9"
server_key_iv: "ec6e856b6d1bc11802482be11942e029"
server_key_version: "encryptionkey@0"
I20260928 10:51:11.359398 11194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:11.386967 11194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:11.387961 11194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:11.388310 11194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:11.411407 11194 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:40127
I20260928 10:51:11.411444 11259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:40127 every 8 connection(s)
I20260928 10:51:11.413619 11194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:11.418694 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:11.420655 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11194
I20260928 10:51:11.420890 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:11.421788 6030 external_mini_cluster.cc:1428] Setting key 73cfb4fb115205335508387d702215e3
I20260928 10:51:11.434897 11260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321: Bootstrap starting.
I20260928 10:51:11.439818 11260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:11.440390 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:11.424683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.5.227.190:40127)
0928 10:51:11.426452 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:11.426469 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:11.426535 (+ 66us) server_negotiation.cc:407] Connection header received
0928 10:51:11.428456 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:11.428485 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:11.429029 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:11.429616 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:11.430549 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:11.432342 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:11.433965 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:11.434866 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:11.435094 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:11.436948 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:11.436962 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:11.436981 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:11.437657 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:11.437766 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:11.437793 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:11.437991 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:11.438262 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 10:51:11.438728 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":539,"threads_started":1}
I20260928 10:51:11.441404 11260 log.cc:824] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:11.445120 11260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321: No bootstrap required, opened a new log
I20260928 10:51:11.445499 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:51:11.454211 11260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05018657ae284e6f86da505e5f4b6321" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40127 } }
I20260928 10:51:11.454695 11260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:11.454835 11260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05018657ae284e6f86da505e5f4b6321, State: Initialized, Role: FOLLOWER
I20260928 10:51:11.455511 11260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05018657ae284e6f86da505e5f4b6321" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40127 } }
I20260928 10:51:11.455835 11260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:11.456013 11260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:11.456220 11260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:11.458734 11260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05018657ae284e6f86da505e5f4b6321" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40127 } }
I20260928 10:51:11.459396 11260 leader_election.cc:302] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05018657ae284e6f86da505e5f4b6321; no voters:
I20260928 10:51:11.460152 11260 leader_election.cc:288] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:11.460351 11265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:11.461782 11265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [term 1 LEADER]: Becoming Leader. State: Replica: 05018657ae284e6f86da505e5f4b6321, State: Running, Role: LEADER
I20260928 10:51:11.462497 11265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05018657ae284e6f86da505e5f4b6321" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40127 } }
I20260928 10:51:11.463250 11260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:11.467291 11266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05018657ae284e6f86da505e5f4b6321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05018657ae284e6f86da505e5f4b6321" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40127 } } }
I20260928 10:51:11.468396 11266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:11.469163 11267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05018657ae284e6f86da505e5f4b6321. Latest consensus state: current_term: 1 leader_uuid: "05018657ae284e6f86da505e5f4b6321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05018657ae284e6f86da505e5f4b6321" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40127 } } }
I20260928 10:51:11.470854 11267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:11.476656 11274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:11.482743 11274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:11.494477 11274 catalog_manager.cc:1347] Generated new cluster ID: cb8dc3c8ce584a47b46e5ab5ff809eaf
I20260928 10:51:11.494619 11274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:11.520972 11274 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:11.522467 11274 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:11.533799 11274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05018657ae284e6f86da505e5f4b6321: Generated new TSK 0
I20260928 10:51:11.535140 11274 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 10:51:11.593192 11291 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 10:51:11.593386 11291 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:51:11.594377 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:11.562598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.5.227.190:40127)
0928 10:51:11.562830 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:11.562837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:11.563566 (+ 729us) server_negotiation.cc:407] Connection header received
0928 10:51:11.564083 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:11.564087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:11.564165 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:11.564339 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:51:11.565101 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:11.565942 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:11.567651 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:11.568259 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:11.568476 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:11.569245 (+ 769us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:51:11.593623 (+ 24378us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 10:51:11.594040 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 10:51:11.594521 11263 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 10:51:11.596470 11290 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:11.562412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40127 (local address 127.0.0.1:55686)
0928 10:51:11.563382 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:11.563405 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:11.563697 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:11.564375 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:11.564380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:11.564395 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:51:11.564736 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:11.564747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:11.566118 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:11.566125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:11.567454 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:11.567469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:11.568552 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:11.568564 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:11.568757 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:11.568837 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:51:11.593971 (+ 25134us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 10:51:11.594383 (+ 412us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.227.190:40127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":520,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.227.190:40127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 10:51:11.598457 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11194
I20260928 10:51:11.599946 11255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:11.702598 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11194
2026-09-28T10:51:11Z chronyd exiting
I20260928 10:51:11.755720 6030 test_util.cc:175] -----------------------------------------------
I20260928 10:51:11.756342 6030 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592619723274-6030-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (904 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 10:51:11.761463 6030 mini_oidc.cc:152] Starting JWKS server
I20260928 10:51:11.763532 6030 webserver.cc:460] Webserver started at https://127.0.0.1:42631/ using document root <none> and password file <none>
I20260928 10:51:11.763861 6030 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:51:11.764674 6030 webserver.cc:460] Webserver started at http://127.0.0.1:36245/ using document root <none> and password file <none>
2026-09-28T10:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:11Z Disabled control of system clock
I20260928 10:51:11.797681 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:43975
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:35767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:43975
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42631/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/testchainca.cert with env {}
W20260928 10:51:12.172596 11298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:12.172994 11298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:12.173169 11298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:12.180946 11298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:12.181133 11298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:12.181238 11298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:12.181334 11298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:12.181401 11298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:51:12.181491 11298 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:51:12.181577 11298 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42631/jwks.json
I20260928 10:51:12.192128 11298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:43975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:43975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42631/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:12.193605 11298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:12.195021 11298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:12.209602 11304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:12.209559 11303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:12.209936 11306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:12.211519 11298 server_base.cc:1034] running on GCE node
I20260928 10:51:12.212700 11298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:12.214996 11298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:12.216303 11298 hybrid_clock.cc:648] HybridClock initialized: now 1790592672216247 us; error 52 us; skew 500 ppm
I20260928 10:51:12.216857 11298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:12.220381 11298 webserver.cc:460] Webserver started at http://127.5.227.190:41817/ using document root <none> and password file <none>
I20260928 10:51:12.221309 11298 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:12.221473 11298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:12.221863 11298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:12.225481 11298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "45fe9cc16bbb48099067fde7e3e82bc7"
format_stamp: "Formatted at 2026-09-28 10:51:12 on dist-test-slave-8w8g"
server_key: "3e1c1ded8fa4cffba11f986090f2bb09"
server_key_iv: "5f48750925aa5acad8a5a1bdc26afd7f"
server_key_version: "encryptionkey@0"
I20260928 10:51:12.226468 11298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "45fe9cc16bbb48099067fde7e3e82bc7"
format_stamp: "Formatted at 2026-09-28 10:51:12 on dist-test-slave-8w8g"
server_key: "3e1c1ded8fa4cffba11f986090f2bb09"
server_key_iv: "5f48750925aa5acad8a5a1bdc26afd7f"
server_key_version: "encryptionkey@0"
I20260928 10:51:12.233681 11298 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260928 10:51:12.238548 11312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:12.240461 11298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:51:12.240697 11298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "45fe9cc16bbb48099067fde7e3e82bc7"
format_stamp: "Formatted at 2026-09-28 10:51:12 on dist-test-slave-8w8g"
server_key: "3e1c1ded8fa4cffba11f986090f2bb09"
server_key_iv: "5f48750925aa5acad8a5a1bdc26afd7f"
server_key_version: "encryptionkey@0"
I20260928 10:51:12.240963 11298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:12.258981 11298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:12.260043 11298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:12.260388 11298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:12.281397 11298 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:43975
I20260928 10:51:12.281419 11363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:43975 every 8 connection(s)
I20260928 10:51:12.282856 11298 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:12.283712 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11298
I20260928 10:51:12.284168 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:12.284862 6030 external_mini_cluster.cc:1428] Setting key 143637c7a58ee5d18b35b24abad89123
I20260928 10:51:12.287604 11364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:12.301854 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:12.286986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.5.227.190:43975)
0928 10:51:12.288218 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:12.288230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:12.288277 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:51:12.290074 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:12.290100 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:12.290654 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:12.291066 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:12.291971 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:12.293407 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:12.295996 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:12.296770 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:12.296932 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:12.298654 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:12.298664 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:12.298691 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:12.299538 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:12.299631 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:12.299656 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:12.299955 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:12.300262 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 10:51:12.300690 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":478,"threads_started":1}
I20260928 10:51:12.302577 11364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7: Bootstrap starting.
I20260928 10:51:12.306960 11364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:12.307173 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:51:12.308321 11364 log.cc:824] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:12.312813 11364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7: No bootstrap required, opened a new log
I20260928 10:51:12.321674 11364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43975 } }
I20260928 10:51:12.322227 11364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:12.322391 11364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fe9cc16bbb48099067fde7e3e82bc7, State: Initialized, Role: FOLLOWER
I20260928 10:51:12.323048 11364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43975 } }
I20260928 10:51:12.323330 11364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:12.323479 11364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:12.323666 11364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:12.326002 11364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43975 } }
I20260928 10:51:12.326613 11364 leader_election.cc:302] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45fe9cc16bbb48099067fde7e3e82bc7; no voters:
I20260928 10:51:12.327304 11364 leader_election.cc:288] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:12.327560 11369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:12.329649 11369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [term 1 LEADER]: Becoming Leader. State: Replica: 45fe9cc16bbb48099067fde7e3e82bc7, State: Running, Role: LEADER
I20260928 10:51:12.330365 11364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:12.330283 11369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43975 } }
I20260928 10:51:12.334391 11371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45fe9cc16bbb48099067fde7e3e82bc7. Latest consensus state: current_term: 1 leader_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43975 } } }
I20260928 10:51:12.334443 11370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fe9cc16bbb48099067fde7e3e82bc7" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43975 } } }
I20260928 10:51:12.335108 11371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:12.335204 11370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:12.336519 11378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:12.342101 11378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:12.353493 11378 catalog_manager.cc:1347] Generated new cluster ID: 13e39eb2c2344656ace59e3278db8122
I20260928 10:51:12.353705 11378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:12.383531 11378 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:12.385371 11378 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:12.394323 11378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45fe9cc16bbb48099067fde7e3e82bc7: Generated new TSK 0
I20260928 10:51:12.395306 11378 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 10:51:12.447281 11394 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 10:51:12.447423 11394 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:51:12.448832 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:12.426603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.5.227.190:43975)
0928 10:51:12.426823 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:12.426831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:12.427340 (+ 509us) server_negotiation.cc:407] Connection header received
0928 10:51:12.427599 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:12.427606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:12.427717 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:12.427916 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:51:12.428457 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:12.429617 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:12.431479 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:12.432238 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:12.432371 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:12.433081 (+ 710us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:51:12.448070 (+ 14989us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 10:51:12.448498 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 10:51:12.448975 11367 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 10:51:12.449065 11393 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:12.426525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:43975 (local address 127.0.0.1:45758)
0928 10:51:12.427222 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:12.427240 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:12.427376 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:12.427951 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:12.427956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:12.427968 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:51:12.428246 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:12.428255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:12.429802 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:12.429808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:12.431320 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:12.431337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:12.432495 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:12.432501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:12.432691 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:12.432741 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:51:12.448343 (+ 15602us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 10:51:12.448653 (+ 310us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.227.190:43975: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":398,"threads_started":1}
I20260928 10:51:12.450201 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11298
I20260928 10:51:12.451956 11359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:12.555138 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11298
2026-09-28T10:51:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (852 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 10:51:12.612792 6030 mini_oidc.cc:152] Starting JWKS server
I20260928 10:51:12.615278 6030 webserver.cc:460] Webserver started at https://127.0.0.1:39935/ using document root <none> and password file <none>
I20260928 10:51:12.615617 6030 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:51:12.616627 6030 webserver.cc:460] Webserver started at http://127.0.0.1:39505/ using document root <none> and password file <none>
2026-09-28T10:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:12Z Disabled control of system clock
I20260928 10:51:12.649398 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:42517
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:42517
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39935/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 10:51:13.021106 11401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:13.021425 11401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:13.021593 11401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:13.029168 11401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:13.029387 11401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:13.029451 11401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:13.029567 11401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:13.029655 11401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:51:13.029748 11401 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:51:13.029820 11401 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39935/jwks.json
I20260928 10:51:13.039806 11401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:42517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39935/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:13.041184 11401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:13.043092 11401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 10:51:13.057037 11401 server_base.cc:1034] running on GCE node
W20260928 10:51:13.057037 11406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:13.057085 11407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:13.056984 11409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:13.058396 11401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:13.060226 11401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:13.061515 11401 hybrid_clock.cc:648] HybridClock initialized: now 1790592673061467 us; error 79 us; skew 500 ppm
I20260928 10:51:13.062091 11401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:13.065042 11401 webserver.cc:460] Webserver started at http://127.5.227.190:40883/ using document root <none> and password file <none>
I20260928 10:51:13.065850 11401 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:13.066006 11401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:13.066370 11401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:13.069610 11401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "a9bcb01319da4ea7808cf1ca926b76a5"
format_stamp: "Formatted at 2026-09-28 10:51:13 on dist-test-slave-8w8g"
server_key: "ed74f18f0e80b8c8bc2be2f3a29b6e29"
server_key_iv: "8c8dfedb9ea0a7b62bbfa4ea8f49dbc1"
server_key_version: "encryptionkey@0"
I20260928 10:51:13.070526 11401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "a9bcb01319da4ea7808cf1ca926b76a5"
format_stamp: "Formatted at 2026-09-28 10:51:13 on dist-test-slave-8w8g"
server_key: "ed74f18f0e80b8c8bc2be2f3a29b6e29"
server_key_iv: "8c8dfedb9ea0a7b62bbfa4ea8f49dbc1"
server_key_version: "encryptionkey@0"
I20260928 10:51:13.077250 11401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 10:51:13.081586 11415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:13.083992 11401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:51:13.084244 11401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "a9bcb01319da4ea7808cf1ca926b76a5"
format_stamp: "Formatted at 2026-09-28 10:51:13 on dist-test-slave-8w8g"
server_key: "ed74f18f0e80b8c8bc2be2f3a29b6e29"
server_key_iv: "8c8dfedb9ea0a7b62bbfa4ea8f49dbc1"
server_key_version: "encryptionkey@0"
I20260928 10:51:13.084514 11401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:13.107707 11401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:13.108801 11401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:13.109181 11401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:13.132287 11401 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:42517
I20260928 10:51:13.132288 11466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:42517 every 8 connection(s)
I20260928 10:51:13.133876 11401 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:13.138157 11467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:13.144306 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11401
I20260928 10:51:13.144515 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:13.145453 6030 external_mini_cluster.cc:1428] Setting key c75edba524aa92e29601c8d988b14403
I20260928 10:51:13.150847 11467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5: Bootstrap starting.
I20260928 10:51:13.154277 11467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:13.155421 11467 log.cc:824] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:13.159435 11467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5: No bootstrap required, opened a new log
I20260928 10:51:13.161638 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:13.147687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.5.227.190:42517)
0928 10:51:13.148953 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:13.148966 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:13.149014 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:51:13.150784 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:13.150820 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:13.151411 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:13.151894 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:13.152498 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.153818 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.155307 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.156404 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.156646 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:13.158274 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:13.158283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:13.158299 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:13.159081 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:13.159180 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:13.159203 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:13.159339 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:13.159930 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 10:51:13.160566 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":400,"threads_started":1}
I20260928 10:51:13.166808 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:51:13.170001 11467 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42517 } }
I20260928 10:51:13.170457 11467 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:13.170590 11467 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bcb01319da4ea7808cf1ca926b76a5, State: Initialized, Role: FOLLOWER
I20260928 10:51:13.171235 11467 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42517 } }
I20260928 10:51:13.171511 11467 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:13.171669 11467 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:13.171878 11467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:13.174515 11467 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42517 } }
I20260928 10:51:13.175168 11467 leader_election.cc:302] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9bcb01319da4ea7808cf1ca926b76a5; no voters:
I20260928 10:51:13.176028 11467 leader_election.cc:288] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:13.176251 11472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:13.177695 11472 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [term 1 LEADER]: Becoming Leader. State: Replica: a9bcb01319da4ea7808cf1ca926b76a5, State: Running, Role: LEADER
I20260928 10:51:13.178503 11472 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42517 } }
I20260928 10:51:13.179450 11467 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:13.181617 11473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42517 } } }
I20260928 10:51:13.182130 11473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:13.184060 11474 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9bcb01319da4ea7808cf1ca926b76a5. Latest consensus state: current_term: 1 leader_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9bcb01319da4ea7808cf1ca926b76a5" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42517 } } }
I20260928 10:51:13.184569 11474 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:13.186787 11480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:13.193082 11480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:13.204463 11480 catalog_manager.cc:1347] Generated new cluster ID: 36e94a093425430e9fb6be8e063cba98
I20260928 10:51:13.204609 11480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:13.229465 11480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:13.231850 11480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:13.242933 11480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9bcb01319da4ea7808cf1ca926b76a5: Generated new TSK 0
I20260928 10:51:13.244067 11480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:51:13.277274 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:13.267437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.5.227.190:42517)
0928 10:51:13.267636 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:13.267645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:13.268098 (+ 453us) server_negotiation.cc:407] Connection header received
0928 10:51:13.268333 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:13.268340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:13.268453 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:13.268655 (+ 202us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:51:13.269069 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.270230 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.271725 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.272737 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.272941 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:13.273632 (+ 691us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:51:13.276558 (+ 2926us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39935/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 10:51:13.276942 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39935/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 10:51:13.277418 11470 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39935/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 10:51:13.279345 11496 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:13.267264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:42517 (local address 127.0.0.1:55236)
0928 10:51:13.267989 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:13.268007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:13.268137 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:13.268652 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:13.268655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:13.268664 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:51:13.268920 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:13.268927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.270408 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.270413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:13.271549 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:13.271563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.273037 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.273043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:13.273208 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:13.273253 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:51:13.276863 (+ 3610us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39935/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 10:51:13.278811 (+ 1948us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.227.190:42517: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39935/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":1521,"thread_start_us":369,"threads_started":1}
I20260928 10:51:13.281030 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11401
I20260928 10:51:13.282526 11462 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 10:51:13.290850 11497 webserver.cc:533] Webserver: sslize failed:
E20260928 10:51:13.290980 11497 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:51:13.379127 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11401
2026-09-28T10:51:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (824 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T10:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:13Z Disabled control of system clock
I20260928 10:51:13.468350 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37125
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:37125
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/testchainca.cert with env {}
W20260928 10:51:13.839700 11502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:13.840142 11502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:13.840240 11502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:13.848369 11502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:13.848507 11502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:13.848544 11502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:13.848594 11502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:13.848642 11502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:51:13.848711 11502 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:51:13.848752 11502 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 10:51:13.859352 11502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:37125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:13.860930 11502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:13.862257 11502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:13.880275 11508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:13.880193 11507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:13.880947 11502 server_base.cc:1034] running on GCE node
W20260928 10:51:13.881196 11510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:13.882501 11502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:13.885027 11502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:13.886611 11502 hybrid_clock.cc:648] HybridClock initialized: now 1790592673886490 us; error 74 us; skew 500 ppm
I20260928 10:51:13.887256 11502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:51:13.890406 11502 webserver.cc:460] Webserver started at http://127.5.227.190:44455/ using document root <none> and password file <none>
I20260928 10:51:13.891659 11502 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:13.892232 11502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:13.892639 11502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:13.896756 11502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "defc98d88b3b4394ba8f987488504865"
format_stamp: "Formatted at 2026-09-28 10:51:13 on dist-test-slave-8w8g"
server_key: "212b8fa8deacb7ac46152011d362da7d"
server_key_iv: "b3a3e1114f6402b4726b7f18a3a9da33"
server_key_version: "encryptionkey@0"
I20260928 10:51:13.897763 11502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "defc98d88b3b4394ba8f987488504865"
format_stamp: "Formatted at 2026-09-28 10:51:13 on dist-test-slave-8w8g"
server_key: "212b8fa8deacb7ac46152011d362da7d"
server_key_iv: "b3a3e1114f6402b4726b7f18a3a9da33"
server_key_version: "encryptionkey@0"
I20260928 10:51:13.904876 11502 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 10:51:13.910235 11516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:13.912389 11502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:51:13.912647 11502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "defc98d88b3b4394ba8f987488504865"
format_stamp: "Formatted at 2026-09-28 10:51:13 on dist-test-slave-8w8g"
server_key: "212b8fa8deacb7ac46152011d362da7d"
server_key_iv: "b3a3e1114f6402b4726b7f18a3a9da33"
server_key_version: "encryptionkey@0"
I20260928 10:51:13.912916 11502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:13.937065 11502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:13.938067 11502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:13.938397 11502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:13.962888 11502 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:37125
I20260928 10:51:13.963243 11567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:37125 every 8 connection(s)
I20260928 10:51:13.964854 11502 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:13.969080 11568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:13.974123 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11502
I20260928 10:51:13.974578 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:13.975554 6030 external_mini_cluster.cc:1428] Setting key 0b01a582f4869d866c3f0a3bf948f057
I20260928 10:51:13.985991 11568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865: Bootstrap starting.
I20260928 10:51:13.990447 11568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:13.992074 11568 log.cc:824] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:13.995668 11568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865: No bootstrap required, opened a new log
I20260928 10:51:13.996309 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:13.978607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.5.227.190:37125)
0928 10:51:13.980860 (+ 2253us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:13.980880 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:13.980945 (+ 65us) server_negotiation.cc:407] Connection header received
0928 10:51:13.984152 (+ 3207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:13.984189 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:13.985056 (+ 867us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:13.985639 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:13.986459 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.988772 (+ 2313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.990343 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:13.991578 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:13.991889 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:13.993344 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:13.993354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:13.993374 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:51:13.993979 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:13.994063 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:51:13.994087 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:13.994232 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:13.994508 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 10:51:13.994865 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":661,"threads_started":1}
I20260928 10:51:14.002121 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:51:14.007488 11568 raft_consensus.cc:348] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defc98d88b3b4394ba8f987488504865" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37125 } }
I20260928 10:51:14.008095 11568 raft_consensus.cc:374] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:14.008296 11568 raft_consensus.cc:729] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc98d88b3b4394ba8f987488504865, State: Initialized, Role: FOLLOWER
I20260928 10:51:14.009186 11568 consensus_queue.cc:260] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defc98d88b3b4394ba8f987488504865" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37125 } }
I20260928 10:51:14.009495 11568 raft_consensus.cc:388] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:14.009665 11568 raft_consensus.cc:482] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:14.009881 11568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:14.012528 11568 raft_consensus.cc:504] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defc98d88b3b4394ba8f987488504865" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37125 } }
I20260928 10:51:14.013196 11568 leader_election.cc:302] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: defc98d88b3b4394ba8f987488504865; no voters:
I20260928 10:51:14.014261 11568 leader_election.cc:288] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:14.014541 11573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:14.015856 11573 raft_consensus.cc:686] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [term 1 LEADER]: Becoming Leader. State: Replica: defc98d88b3b4394ba8f987488504865, State: Running, Role: LEADER
I20260928 10:51:14.016952 11573 consensus_queue.cc:237] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defc98d88b3b4394ba8f987488504865" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37125 } }
I20260928 10:51:14.017459 11568 sys_catalog.cc:564] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:14.021607 11575 sys_catalog.cc:455] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader defc98d88b3b4394ba8f987488504865. Latest consensus state: current_term: 1 leader_uuid: "defc98d88b3b4394ba8f987488504865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defc98d88b3b4394ba8f987488504865" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37125 } } }
I20260928 10:51:14.021761 11574 sys_catalog.cc:455] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "defc98d88b3b4394ba8f987488504865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defc98d88b3b4394ba8f987488504865" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37125 } } }
I20260928 10:51:14.022396 11575 sys_catalog.cc:458] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:14.022480 11574 sys_catalog.cc:458] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:14.024262 11581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:14.032001 11581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:14.043923 11581 catalog_manager.cc:1347] Generated new cluster ID: 13615f40d0e64b1eb25880919f4c131f
I20260928 10:51:14.044070 11581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:14.084066 11581 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:14.085965 11581 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:14.094254 11581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P defc98d88b3b4394ba8f987488504865: Generated new TSK 0
I20260928 10:51:14.095255 11581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:51:14.187976 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:14.109067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.5.227.190:37125)
0928 10:51:14.109294 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:14.109302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:14.109737 (+ 435us) server_negotiation.cc:407] Connection header received
0928 10:51:14.110032 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:14.110039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:14.110140 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:14.110342 (+ 202us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:51:14.110965 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:14.112317 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:14.114126 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:14.114957 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:14.115126 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:14.115994 (+ 868us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:51:14.187180 (+ 71186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 10:51:14.187618 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51686: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 10:51:14.188131 11571 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51686: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 10:51:14.188535 11597 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:14.108877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37125 (local address 127.0.0.1:51686)
0928 10:51:14.109593 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:14.109610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:14.109809 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:14.110404 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:14.110409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:14.110420 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:51:14.110734 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:14.110744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:14.112523 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:14.112528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:14.113925 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:14.113944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:14.115212 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:14.115217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:14.115382 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:14.115426 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:51:14.187540 (+ 72114us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 10:51:14.187942 (+ 402us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.227.190:37125: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":396,"threads_started":1}
I20260928 10:51:14.190088 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11502
I20260928 10:51:14.191886 11563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:14.309405 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11502
2026-09-28T10:51:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (913 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (454 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (508 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:51:15 dist-test-slave-8w8g krb5kdc[11612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:51:15 dist-test-slave-8w8g krb5kdc[11612](info): set up 2 sockets
Sep 28 10:51:15 dist-test-slave-8w8g krb5kdc[11612](info): commencing operation
krb5kdc: starting...
W20260928 10:51:16.441222 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:51:16 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:51:16.722309 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:38643
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:17.133612 11625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:17.134042 11625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:17.134225 11625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:17.143596 11625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:17.143810 11625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:17.143916 11625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:17.144042 11625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:17.144105 11625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:17.156949 11625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:38643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:17.158942 11625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:17.160619 11625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:17.178076 11631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:17.178099 11633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:17.179476 11630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:17.180981 11625 server_base.cc:1034] running on GCE node
I20260928 10:51:17.181975 11625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:17.183956 11625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:17.185253 11625 hybrid_clock.cc:648] HybridClock initialized: now 1790592677185213 us; error 111 us; skew 500 ppm
I20260928 10:51:17.185708 11625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:17 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592677, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:17.190240 11625 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:51:17.192395 11625 webserver.cc:460] Webserver started at http://127.5.227.190:38053/ using document root <none> and password file <none>
I20260928 10:51:17.193238 11625 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:17.193410 11625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:17.193794 11625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:17.198525 11625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "468eac4becb1480ab01412b37c6dde89"
format_stamp: "Formatted at 2026-09-28 10:51:17 on dist-test-slave-8w8g"
server_key: "38ef2bd47525928e04348ab9e07ada44"
server_key_iv: "bce0f981538a5aa54ca08f11037b351c"
server_key_version: "encryptionkey@0"
I20260928 10:51:17.199579 11625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "468eac4becb1480ab01412b37c6dde89"
format_stamp: "Formatted at 2026-09-28 10:51:17 on dist-test-slave-8w8g"
server_key: "38ef2bd47525928e04348ab9e07ada44"
server_key_iv: "bce0f981538a5aa54ca08f11037b351c"
server_key_version: "encryptionkey@0"
I20260928 10:51:17.206399 11625 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:51:17.211460 11640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:17.213568 11625 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:51:17.213781 11625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "468eac4becb1480ab01412b37c6dde89"
format_stamp: "Formatted at 2026-09-28 10:51:17 on dist-test-slave-8w8g"
server_key: "38ef2bd47525928e04348ab9e07ada44"
server_key_iv: "bce0f981538a5aa54ca08f11037b351c"
server_key_version: "encryptionkey@0"
I20260928 10:51:17.213986 11625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:17.241036 11625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:17.246925 11625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:17.247303 11625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:17.270010 11625 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:38643
I20260928 10:51:17.270078 11691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:38643 every 8 connection(s)
I20260928 10:51:17.271577 11625 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:17.276180 11692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:17.280381 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11625
I20260928 10:51:17.280642 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:17.281452 6030 external_mini_cluster.cc:1428] Setting key 12c501fe5f0fb8a42e1ea093ca50f06e
I20260928 10:51:17.289718 11692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89: Bootstrap starting.
I20260928 10:51:17.294217 11692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:17.295645 11692 log.cc:824] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:17.300109 11692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89: No bootstrap required, opened a new log
Sep 28 10:51:17 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:17.308575 11692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468eac4becb1480ab01412b37c6dde89" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38643 } }
I20260928 10:51:17.309077 11692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:17.309232 11692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468eac4becb1480ab01412b37c6dde89, State: Initialized, Role: FOLLOWER
I20260928 10:51:17.309957 11692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468eac4becb1480ab01412b37c6dde89" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38643 } }
I20260928 10:51:17.310276 11692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:17.310443 11692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:17.310642 11692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:17.311018 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:17.284772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.5.227.190:38643)
0928 10:51:17.285695 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:17.285709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:17.285762 (+ 53us) server_negotiation.cc:407] Connection header received
0928 10:51:17.287692 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:17.287722 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:17.288321 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:17.288709 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:17.290761 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:17.292434 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:17.294511 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:17.295385 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:17.295571 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:17.301921 (+ 6350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:17.301929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:17.301948 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:17.301996 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:17.305808 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:17.306754 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:17.306766 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:17.306775 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:17.306999 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:17.307483 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:17.307488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:17.307492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:17.308243 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:17.308598 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:17.309164 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 10:51:17.309727 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":427,"threads_started":1}
I20260928 10:51:17.313352 11692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468eac4becb1480ab01412b37c6dde89" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38643 } }
I20260928 10:51:17.313998 11692 leader_election.cc:302] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 468eac4becb1480ab01412b37c6dde89; no voters:
I20260928 10:51:17.314770 11692 leader_election.cc:288] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:17.314869 11697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:17.316088 11697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [term 1 LEADER]: Becoming Leader. State: Replica: 468eac4becb1480ab01412b37c6dde89, State: Running, Role: LEADER
I20260928 10:51:17.316965 11697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468eac4becb1480ab01412b37c6dde89" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38643 } }
I20260928 10:51:17.317848 11692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:17.320113 11699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468eac4becb1480ab01412b37c6dde89. Latest consensus state: current_term: 1 leader_uuid: "468eac4becb1480ab01412b37c6dde89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468eac4becb1480ab01412b37c6dde89" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38643 } } }
I20260928 10:51:17.320264 11698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "468eac4becb1480ab01412b37c6dde89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468eac4becb1480ab01412b37c6dde89" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38643 } } }
I20260928 10:51:17.320681 11699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:17.320935 11698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:17.324816 11703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:17.332185 11703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:17.344512 11703 catalog_manager.cc:1347] Generated new cluster ID: 919d40fbbd0447179d553974d47370a3
I20260928 10:51:17.344755 11703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:17.409106 11703 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:17.411085 11703 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:17.428828 11703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 468eac4becb1480ab01412b37c6dde89: Generated new TSK 0
I20260928 10:51:17.430080 11703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:51:17.510083 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38643
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:17.948725 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:17.949182 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:17.949455 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:17.958396 11718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:17.958534 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:17.958731 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:17.972025 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:17.974231 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:17.975888 11718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:17.992851 11723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:17.994500 11724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:17.994501 11726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:17.996341 11718 server_base.cc:1034] running on GCE node
I20260928 10:51:17.997079 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:17.998178 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:17.999675 11718 hybrid_clock.cc:648] HybridClock initialized: now 1790592677999570 us; error 84 us; skew 500 ppm
I20260928 10:51:18.000146 11718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:17 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592677, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:18.003847 11718 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:51:18.005797 11718 webserver.cc:460] Webserver started at http://127.5.227.129:46329/ using document root <none> and password file <none>
I20260928 10:51:18.006590 11718 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:18.006708 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:18.006982 11718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:18.010704 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "864a994988224c9f8493934b82e59695"
format_stamp: "Formatted at 2026-09-28 10:51:18 on dist-test-slave-8w8g"
server_key: "3ec468b6c411412bb76dcca1e054be83"
server_key_iv: "bc07e136f0964e8f28252265a43b0e36"
server_key_version: "encryptionkey@0"
I20260928 10:51:18.011657 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "864a994988224c9f8493934b82e59695"
format_stamp: "Formatted at 2026-09-28 10:51:18 on dist-test-slave-8w8g"
server_key: "3ec468b6c411412bb76dcca1e054be83"
server_key_iv: "bc07e136f0964e8f28252265a43b0e36"
server_key_version: "encryptionkey@0"
I20260928 10:51:18.021122 11718 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260928 10:51:18.027045 11733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:18.029356 11718 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:51:18.029624 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "864a994988224c9f8493934b82e59695"
format_stamp: "Formatted at 2026-09-28 10:51:18 on dist-test-slave-8w8g"
server_key: "3ec468b6c411412bb76dcca1e054be83"
server_key_iv: "bc07e136f0964e8f28252265a43b0e36"
server_key_version: "encryptionkey@0"
I20260928 10:51:18.029879 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:18.059232 11718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:18.065130 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:18.065483 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:18.066682 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:18.068521 11718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:18.068866 11718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:18.069083 11718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:18.069180 11718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:18.115036 11718 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:42791
I20260928 10:51:18.115072 11845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:42791 every 8 connection(s)
I20260928 10:51:18.116515 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:18.122663 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11718
I20260928 10:51:18.122915 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:51:18.123718 6030 external_mini_cluster.cc:1428] Setting key 14ee429cee3b6b019d47e68bca7e94a9
Sep 28 10:51:18 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592677, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:18.149616 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:18.119682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:53287 (local address 127.5.227.190:38643)
0928 10:51:18.120261 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:18.120268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:18.121525 (+ 1257us) server_negotiation.cc:407] Connection header received
0928 10:51:18.124428 (+ 2903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:18.124434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:18.124517 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:18.124820 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:18.127661 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.129239 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.131087 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.131908 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.132070 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:18.139141 (+ 7071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:18.139147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:18.139151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:18.139201 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:18.143521 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:18.145559 (+ 2038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:18.145565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:18.145567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:18.145753 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:18.146809 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:18.146814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:18.146817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:18.147292 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:18.147436 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:18.148916 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0928 10:51:18.149138 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":288,"threads_started":1}
I20260928 10:51:18.151023 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:18.120398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38643 (local address 127.5.227.129:53287)
0928 10:51:18.121242 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:18.121299 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:18.123883 (+ 2584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:18.125195 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:18.125209 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:18.126354 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:18.127413 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:18.127441 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.129454 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.129458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:18.130868 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:18.130885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.132260 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.132267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:18.132510 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:18.134097 (+ 1587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:18.134133 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:18.138706 (+ 4573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:18.143962 (+ 5256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:18.143977 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:18.144003 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:18.145330 (+ 1327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:18.145991 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:18.146000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:18.146006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:18.146488 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:18.147530 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:18.147540 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:18.148252 (+ 712us) client_negotiation.cc:769] Sending connection context
0928 10:51:18.148661 (+ 409us) client_negotiation.cc:240] Negotiation successful
0928 10:51:18.149351 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":340,"threads_started":1}
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:51:18.153461 11846 heartbeater.cc:347] Connected to a master server at 127.5.227.190:38643
I20260928 10:51:18.153820 11846 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:18.154855 11846 heartbeater.cc:511] Master 127.5.227.190:38643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:51:18.158984 11657 ts_manager.cc:194] Registered new tserver with Master: 864a994988224c9f8493934b82e59695 (127.5.227.129:42791)
I20260928 10:51:18.162390 11657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:53287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:51:18.204432 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38643
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:18.611578 11852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:18.612026 11852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:18.612215 11852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:18.620388 11852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:18.620558 11852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:18.620859 11852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:18.634080 11852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:18.635666 11852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:18.637626 11852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:18.654851 11860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:18.654958 11858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:18.654980 11857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:18.655828 11852 server_base.cc:1034] running on GCE node
I20260928 10:51:18.656785 11852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:18.658066 11852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:18.659340 11852 hybrid_clock.cc:648] HybridClock initialized: now 1790592678659298 us; error 37 us; skew 500 ppm
I20260928 10:51:18.659967 11852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:18 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592678, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:18.664254 11852 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:51:18.666226 11852 webserver.cc:460] Webserver started at http://127.5.227.130:38595/ using document root <none> and password file <none>
I20260928 10:51:18.667057 11852 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:18.667465 11852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:18.667950 11852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:18.671550 11852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "9c020bac49714d578d13e8c575213e62"
format_stamp: "Formatted at 2026-09-28 10:51:18 on dist-test-slave-8w8g"
server_key: "9ece9ced4ff93da47c6015de7412f3cc"
server_key_iv: "42bbe7cd9481fcc218a397ebce857c3b"
server_key_version: "encryptionkey@0"
I20260928 10:51:18.673020 11852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "9c020bac49714d578d13e8c575213e62"
format_stamp: "Formatted at 2026-09-28 10:51:18 on dist-test-slave-8w8g"
server_key: "9ece9ced4ff93da47c6015de7412f3cc"
server_key_iv: "42bbe7cd9481fcc218a397ebce857c3b"
server_key_version: "encryptionkey@0"
I20260928 10:51:18.680318 11852 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 10:51:18.684937 11867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:18.687306 11852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:51:18.687570 11852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "9c020bac49714d578d13e8c575213e62"
format_stamp: "Formatted at 2026-09-28 10:51:18 on dist-test-slave-8w8g"
server_key: "9ece9ced4ff93da47c6015de7412f3cc"
server_key_iv: "42bbe7cd9481fcc218a397ebce857c3b"
server_key_version: "encryptionkey@0"
I20260928 10:51:18.687978 11852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:18.714609 11852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:18.720594 11852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:18.720929 11852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:18.722013 11852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:18.723618 11852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:18.723734 11852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:18.723922 11852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:18.724001 11852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:18.766462 11852 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:38133
I20260928 10:51:18.766570 11979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:38133 every 8 connection(s)
I20260928 10:51:18.767941 11852 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:18.772891 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11852
I20260928 10:51:18.773137 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:51:18.773942 6030 external_mini_cluster.cc:1428] Setting key b4e4b6c765d3178e564a3ff45e38d9e6
Sep 28 10:51:18 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592678, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:18.796096 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:18.771154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:52381 (local address 127.5.227.190:38643)
0928 10:51:18.771694 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:18.771698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:18.773702 (+ 2004us) server_negotiation.cc:407] Connection header received
0928 10:51:18.776099 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:18.776105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:18.776259 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:18.776392 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:18.778316 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.779053 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.781503 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.781999 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.782163 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:18.786938 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:18.786949 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:18.786957 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:18.787013 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:18.791317 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:18.792792 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:18.792800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:18.792804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:18.793000 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:18.793614 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:18.793622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:18.793628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:18.794135 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:18.794326 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:18.795279 (+ 953us) server_negotiation.cc:299] Negotiation successful
0928 10:51:18.795576 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20260928 10:51:18.797057 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:18.771887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38643 (local address 127.5.227.130:52381)
0928 10:51:18.773085 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:18.773165 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:18.775616 (+ 2451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:18.776780 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:18.776790 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:18.777399 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:18.778089 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:18.778111 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.779264 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.779269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:18.781258 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:18.781279 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:18.782328 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:18.782335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:18.782571 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:18.783683 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:18.783713 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:18.786557 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:18.791576 (+ 5019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:18.791685 (+ 109us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:18.791710 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:18.792470 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:18.793204 (+ 734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:18.793209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:18.793211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:18.793435 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:18.794320 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:18.794327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:18.794746 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 10:51:18.795024 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 10:51:18.795511 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":466,"threads_started":1}
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:51:18.799196 11980 heartbeater.cc:347] Connected to a master server at 127.5.227.190:38643
I20260928 10:51:18.799559 11980 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:18.800426 11980 heartbeater.cc:511] Master 127.5.227.190:38643 requested a full tablet report, sending...
I20260928 10:51:18.802381 11657 ts_manager.cc:194] Registered new tserver with Master: 9c020bac49714d578d13e8c575213e62 (127.5.227.130:38133)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:51:18.804203 11657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:52381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:51:18.842288 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38643
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:19.166858 11846 heartbeater.cc:503] Master 127.5.227.190:38643 was elected leader, sending a full tablet report...
W20260928 10:51:19.225016 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:19.225358 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:19.225524 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:19.234617 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:19.234776 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:19.234974 11986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:19.250231 11986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:19.251981 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:19.253343 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:19.271791 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:19.272811 11994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:19.274181 11986 server_base.cc:1034] running on GCE node
W20260928 10:51:19.274681 11992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:19.275637 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:19.277573 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:19.279167 11986 hybrid_clock.cc:648] HybridClock initialized: now 1790592679279038 us; error 116 us; skew 500 ppm
I20260928 10:51:19.279579 11986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.283428 11986 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:51:19.285641 11986 webserver.cc:460] Webserver started at http://127.5.227.131:45515/ using document root <none> and password file <none>
I20260928 10:51:19.286725 11986 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:19.286921 11986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:19.287283 11986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:19.291543 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "4de5abef031848f2b34c2d869492b82b"
format_stamp: "Formatted at 2026-09-28 10:51:19 on dist-test-slave-8w8g"
server_key: "a8b1142828a85c86ba1dcef47b47a873"
server_key_iv: "f547b0a36b63d62a0e04a8cc5b9892f2"
server_key_version: "encryptionkey@0"
I20260928 10:51:19.293274 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "4de5abef031848f2b34c2d869492b82b"
format_stamp: "Formatted at 2026-09-28 10:51:19 on dist-test-slave-8w8g"
server_key: "a8b1142828a85c86ba1dcef47b47a873"
server_key_iv: "f547b0a36b63d62a0e04a8cc5b9892f2"
server_key_version: "encryptionkey@0"
I20260928 10:51:19.300891 11986 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 10:51:19.306768 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:19.309031 11986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:51:19.309394 11986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "4de5abef031848f2b34c2d869492b82b"
format_stamp: "Formatted at 2026-09-28 10:51:19 on dist-test-slave-8w8g"
server_key: "a8b1142828a85c86ba1dcef47b47a873"
server_key_iv: "f547b0a36b63d62a0e04a8cc5b9892f2"
server_key_version: "encryptionkey@0"
I20260928 10:51:19.310164 11986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:19.341037 11986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:19.346923 11986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:19.347301 11986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:19.348464 11986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:19.350118 11986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:19.350257 11986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:19.350430 11986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:19.350505 11986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:19.399813 11986 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:39461
I20260928 10:51:19.399582 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:39461 every 8 connection(s)
I20260928 10:51:19.401639 11986 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:19.411687 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 11986
I20260928 10:51:19.411996 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:51:19.412569 6030 external_mini_cluster.cc:1428] Setting key 829b3e02028276ac9037e4de516d8259
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:19.430495 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.405118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:60421 (local address 127.5.227.190:38643)
0928 10:51:19.405900 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.405905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.407506 (+ 1601us) server_negotiation.cc:407] Connection header received
0928 10:51:19.409981 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.409988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.410112 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.410246 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.412609 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.413420 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.415340 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.415985 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.416136 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.420852 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:19.420859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:19.420863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:19.420913 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:19.425441 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.426855 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.426866 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.426868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.427058 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.427850 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.427857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.427860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.428427 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:19.428760 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:19.429712 (+ 952us) server_negotiation.cc:299] Negotiation successful
0928 10:51:19.430006 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":561,"threads_started":1}
I20260928 10:51:19.432308 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.406060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38643 (local address 127.5.227.131:60421)
0928 10:51:19.407084 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:19.407140 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:19.409593 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:19.410547 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:19.410557 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:19.411430 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:19.412355 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:19.412380 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.413623 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.413627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:19.415119 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:19.415178 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.416268 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.416277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:19.416740 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.417536 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:19.417562 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:19.420492 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:19.426046 (+ 5554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:19.426062 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:19.426078 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:19.426680 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:19.427188 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:19.427194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:19.427198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:19.427483 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:19.428626 (+ 1143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:19.428640 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:19.429261 (+ 621us) client_negotiation.cc:769] Sending connection context
0928 10:51:19.429609 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 10:51:19.430316 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":500,"threads_started":1}
I20260928 10:51:19.436007 12114 heartbeater.cc:347] Connected to a master server at 127.5.227.190:38643
I20260928 10:51:19.436425 12114 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:19.437531 12114 heartbeater.cc:511] Master 127.5.227.190:38643 requested a full tablet report, sending...
I20260928 10:51:19.439555 11657 ts_manager.cc:194] Registered new tserver with Master: 4de5abef031848f2b34c2d869492b82b (127.5.227.131:39461)
I20260928 10:51:19.440526 11657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:60421
I20260928 10:51:19.444304 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/test-admin.keytab.
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.489490 6030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 10:51:19.497165 12122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:19.493457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.5.227.129:42791)
0928 10:51:19.494351 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.494358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.494388 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:51:19.494724 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.494732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.494960 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.495138 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.496766 (+ 1628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54760: BlockingRecv error: recv got EOF from 127.0.0.1:54760 (error 108)
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":478,"threads_started":1}
W20260928 10:51:19.497742 6030 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5cc' of type FILE
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.499600 6030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.501305 6030 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:51:19.522745 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.503415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.5.227.129:42791)
0928 10:51:19.503809 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.503817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.504179 (+ 362us) server_negotiation.cc:407] Connection header received
0928 10:51:19.504407 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.504411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.504479 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.504597 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.505756 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.506784 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.508786 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.509433 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.509837 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.514194 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:19.514211 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:19.514228 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:19.514281 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:19.518512 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.519560 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.519569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.519572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.519851 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.520500 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.520505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.520508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.521444 (+ 936us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:19.521849 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:19.522207 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 10:51:19.522442 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
W20260928 10:51:19.531385 12122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:19.528648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.5.227.129:42791)
0928 10:51:19.528967 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.528974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.529305 (+ 331us) server_negotiation.cc:407] Connection header received
0928 10:51:19.529551 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.529555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.529760 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.529917 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.531080 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54784: BlockingRecv error: recv got EOF from 127.0.0.1:54784 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
W20260928 10:51:19.531991 6030 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5cc' of type FILE
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.534191 6030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.535929 6030 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:51:19.556787 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.538105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.5.227.129:42791)
0928 10:51:19.538433 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.538438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.538578 (+ 140us) server_negotiation.cc:407] Connection header received
0928 10:51:19.538958 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.538963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.539064 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.539171 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.541104 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.541831 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.543388 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.543911 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.544049 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.549156 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:19.549169 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:19.549173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:19.549234 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:19.552455 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.554083 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.554089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.554091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.554346 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.555008 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.555013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.555015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.555433 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:19.555698 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:19.556320 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 10:51:19.556517 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
W20260928 10:51:19.564224 12122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:19.562123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.5.227.129:42791)
0928 10:51:19.562373 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.562380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.562593 (+ 213us) server_negotiation.cc:407] Connection header received
0928 10:51:19.562820 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.562827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.562912 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.563037 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.563997 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54798: BlockingRecv error: recv got EOF from 127.0.0.1:54798 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 10:51:19.564782 6030 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592619723274-6030-0/krb5kdc/krb5cc' of type FILE
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.567135 6030 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:19.568729 6030 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:51:19.593485 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.571943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.5.227.129:42791)
0928 10:51:19.572529 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.572550 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.572578 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:51:19.572870 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.572877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.573005 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.573151 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.574873 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.575957 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.578248 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.579061 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.579227 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.584617 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:19.584623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:19.584627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:19.584675 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:19.588292 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.589427 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.589434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.589438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.589979 (+ 541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.590698 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.590704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.590708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.591528 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:19.592428 (+ 900us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:19.592862 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 10:51:19.593077 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260928 10:51:19.596378 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11718
I20260928 10:51:19.614509 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.597927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.5.227.129:42791)
0928 10:51:19.598117 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.598129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.598275 (+ 146us) server_negotiation.cc:407] Connection header received
0928 10:51:19.598508 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.598512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.598681 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.598785 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.600240 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.601170 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.602570 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.603366 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.603491 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.606171 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:19.606177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:19.606180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:19.606222 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:19.610183 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.611597 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.611602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.611604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.611991 (+ 387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.612511 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.612515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.612518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.612872 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:19.613097 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:19.613891 (+ 794us) server_negotiation.cc:299] Negotiation successful
0928 10:51:19.614102 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:51:19.615520 11841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:19.725870 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11718
I20260928 10:51:19.746367 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11852
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:51:19.769126 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.747722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.5.227.130:38133)
0928 10:51:19.748508 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.748516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.748546 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:51:19.748610 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.748616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.748848 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.749071 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.750206 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.751810 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.754430 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.755376 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.755581 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.760345 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:19.760372 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:19.760391 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:19.760439 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:19.763881 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.764933 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.764942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.764944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.765117 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.766298 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.766304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.766306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.767065 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:19.767967 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:19.768429 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 10:51:19.768752 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":353,"threads_started":1}
I20260928 10:51:19.770687 11975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:19.867201 11980 heartbeater.cc:503] Master 127.5.227.190:38643 was elected leader, sending a full tablet report...
I20260928 10:51:19.868374 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11852
W20260928 10:51:19.886422 11644 connection.cc:535] server connection from 127.5.227.130:52381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:51:19.886818 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11986
Sep 28 10:51:19 dist-test-slave-8w8g krb5kdc[11612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:51:19.906656 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:19.888040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.5.227.131:39461)
0928 10:51:19.888652 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:19.888660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:19.888693 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:51:19.888760 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:19.888766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:19.888993 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:19.889254 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:19.890434 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.891636 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.893245 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:19.893895 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:19.894095 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:19.899195 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:19.899217 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:19.899234 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:19.899293 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:19.903360 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.904414 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.904423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.904426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.904612 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:19.905093 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:19.905097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:19.905100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:19.905486 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:19.905874 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:19.906126 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 10:51:19.906348 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":295,"threads_started":1}
I20260928 10:51:19.908295 12109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:19.996255 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11986
I20260928 10:51:20.014043 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 11625
I20260928 10:51:20.015349 11687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:20.123842 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 11625
2026-09-28T10:51:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4853 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:51:20 dist-test-slave-8w8g krb5kdc[12142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:51:20 dist-test-slave-8w8g krb5kdc[12142](info): set up 2 sockets
Sep 28 10:51:20 dist-test-slave-8w8g krb5kdc[12142](info): commencing operation
krb5kdc: starting...
W20260928 10:51:21.317823 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:51:21 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab.
Entry for principal oryx/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab.
I20260928 10:51:21.574146 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36225
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab
--principal=oryx/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:21.946547 12155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:21.946928 12155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:21.947050 12155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:21.954638 12155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:21.954785 12155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:21.954847 12155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:21.954905 12155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:21.954941 12155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:21.965593 12155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab
--principal=oryx/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:21.967067 12155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:21.968523 12155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:21.983872 12160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:21.983716 12161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:21.983767 12163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:21.985004 12155 server_base.cc:1034] running on GCE node
I20260928 10:51:21.985916 12155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:21.987653 12155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:21.988974 12155 hybrid_clock.cc:648] HybridClock initialized: now 1790592681988915 us; error 53 us; skew 500 ppm
I20260928 10:51:21.989398 12155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:21 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:21.998755 12155 init.cc:377] Logged in from keytab as oryx/127.5.227.190@KRBTEST.COM (short username oryx)
I20260928 10:51:22.000952 12155 webserver.cc:460] Webserver started at http://127.5.227.190:33165/ using document root <none> and password file <none>
I20260928 10:51:22.002190 12155 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:22.002341 12155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:22.002784 12155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:22.006261 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "bbaae6cf9d0245659e76c2d34361d683"
format_stamp: "Formatted at 2026-09-28 10:51:21 on dist-test-slave-8w8g"
server_key: "5febcc59f7db89d034c884742b6d8c59"
server_key_iv: "6fbcd5ac7fa80556dcc1251b922963b0"
server_key_version: "encryptionkey@0"
I20260928 10:51:22.007210 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "bbaae6cf9d0245659e76c2d34361d683"
format_stamp: "Formatted at 2026-09-28 10:51:21 on dist-test-slave-8w8g"
server_key: "5febcc59f7db89d034c884742b6d8c59"
server_key_iv: "6fbcd5ac7fa80556dcc1251b922963b0"
server_key_version: "encryptionkey@0"
I20260928 10:51:22.013841 12155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:51:22.019686 12170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:22.022373 12155 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260928 10:51:22.022984 12155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "bbaae6cf9d0245659e76c2d34361d683"
format_stamp: "Formatted at 2026-09-28 10:51:21 on dist-test-slave-8w8g"
server_key: "5febcc59f7db89d034c884742b6d8c59"
server_key_iv: "6fbcd5ac7fa80556dcc1251b922963b0"
server_key_version: "encryptionkey@0"
I20260928 10:51:22.023324 12155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:22.040928 12155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:22.046868 12155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:22.047194 12155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:22.070647 12155 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:36225
I20260928 10:51:22.070854 12221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:36225 every 8 connection(s)
I20260928 10:51:22.072239 12155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:22.076777 12222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:22.079973 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 12155
I20260928 10:51:22.080284 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:22.081012 6030 external_mini_cluster.cc:1428] Setting key 75c1e673ddf1a3fa1ee2ae5e0147a673
I20260928 10:51:22.091799 12222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: Bootstrap starting.
I20260928 10:51:22.096805 12222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:22.098403 12222 log.cc:824] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:51:22 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:22.102770 12222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: No bootstrap required, opened a new log
I20260928 10:51:22.110886 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:22.083913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.5.227.190:36225)
0928 10:51:22.084783 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:22.084799 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:22.084850 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:51:22.087045 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:22.087084 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:22.087838 (+ 754us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:22.088406 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:22.090186 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:22.092320 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:22.093833 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:22.095203 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:22.095452 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:22.100735 (+ 5283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:22.100746 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:22.100764 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:22.100809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:22.105098 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:22.106292 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:22.106300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:22.106311 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:22.106679 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:22.107213 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:22.107217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:22.107220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:22.108086 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:22.108442 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:22.109019 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 10:51:22.109646 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":354,"threads_started":1}
I20260928 10:51:22.112805 12222 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } }
I20260928 10:51:22.113240 12222 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:22.113366 12222 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaae6cf9d0245659e76c2d34361d683, State: Initialized, Role: FOLLOWER
I20260928 10:51:22.113993 12222 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } }
I20260928 10:51:22.114343 12222 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:22.114539 12222 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:22.114843 12222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:22.117240 12222 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } }
I20260928 10:51:22.118028 12222 leader_election.cc:302] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbaae6cf9d0245659e76c2d34361d683; no voters:
I20260928 10:51:22.118981 12222 leader_election.cc:288] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:22.119225 12227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:22.120736 12227 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 1 LEADER]: Becoming Leader. State: Replica: bbaae6cf9d0245659e76c2d34361d683, State: Running, Role: LEADER
I20260928 10:51:22.121342 12227 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } }
I20260928 10:51:22.121912 12222 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:22.124333 12229 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbaae6cf9d0245659e76c2d34361d683. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } }
I20260928 10:51:22.124854 12229 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:22.125252 12228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } }
I20260928 10:51:22.125703 12228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:22.127214 12232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:22.134723 12232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:22.147940 12232 catalog_manager.cc:1347] Generated new cluster ID: ecb34afe09b145fcb8491764a29d4a70
I20260928 10:51:22.148097 12232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:22.172457 12232 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:22.174328 12232 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:22.182260 12232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: Generated new TSK 0
I20260928 10:51:22.183473 12232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab.
Entry for principal oryx/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab.
I20260928 10:51:22.265416 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36225
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab
--principal=oryx/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:22.665378 12248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:22.665764 12248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:22.665928 12248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:22.673846 12248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:22.674034 12248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:22.674204 12248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:22.685637 12248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab
--principal=oryx/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:22.687109 12248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:22.688558 12248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:22.707701 12254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:22.707763 12256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:22.708021 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:22.707845 12248 server_base.cc:1034] running on GCE node
I20260928 10:51:22.708978 12248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:22.710094 12248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:22.711345 12248 hybrid_clock.cc:648] HybridClock initialized: now 1790592682711289 us; error 42 us; skew 500 ppm
I20260928 10:51:22.711905 12248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:22 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:22.716802 12248 init.cc:377] Logged in from keytab as oryx/127.5.227.129@KRBTEST.COM (short username oryx)
I20260928 10:51:22.718654 12248 webserver.cc:460] Webserver started at http://127.5.227.129:36041/ using document root <none> and password file <none>
I20260928 10:51:22.719460 12248 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:22.719585 12248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:22.720002 12248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:22.723935 12248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "569f926721f94d0d9d6102ae8eba1011"
format_stamp: "Formatted at 2026-09-28 10:51:22 on dist-test-slave-8w8g"
server_key: "fa62b3c284efdad0a771ae3a95e3d237"
server_key_iv: "5fea4b74b168c9bcb3dbac0cb0b0acb8"
server_key_version: "encryptionkey@0"
I20260928 10:51:22.724922 12248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "569f926721f94d0d9d6102ae8eba1011"
format_stamp: "Formatted at 2026-09-28 10:51:22 on dist-test-slave-8w8g"
server_key: "fa62b3c284efdad0a771ae3a95e3d237"
server_key_iv: "5fea4b74b168c9bcb3dbac0cb0b0acb8"
server_key_version: "encryptionkey@0"
I20260928 10:51:22.732616 12248 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260928 10:51:22.737157 12263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:22.739302 12248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:51:22.739589 12248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "569f926721f94d0d9d6102ae8eba1011"
format_stamp: "Formatted at 2026-09-28 10:51:22 on dist-test-slave-8w8g"
server_key: "fa62b3c284efdad0a771ae3a95e3d237"
server_key_iv: "5fea4b74b168c9bcb3dbac0cb0b0acb8"
server_key_version: "encryptionkey@0"
I20260928 10:51:22.739853 12248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:22.773160 12248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:22.779214 12248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:22.779575 12248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:22.780870 12248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:22.782423 12248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:22.782531 12248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:22.782675 12248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:22.782752 12248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:22.823398 12248 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:37621
I20260928 10:51:22.823509 12375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:37621 every 8 connection(s)
I20260928 10:51:22.824910 12248 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:22.834389 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 12248
I20260928 10:51:22.834663 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:51:22.835443 6030 external_mini_cluster.cc:1428] Setting key d04899e8aec5f0fa8d5b8410bfc9f81d
Sep 28 10:51:22 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:22.850793 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:22.828073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:35389 (local address 127.5.227.190:36225)
0928 10:51:22.828694 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:22.828704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:22.829820 (+ 1116us) server_negotiation.cc:407] Connection header received
0928 10:51:22.831865 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:22.831872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:22.831977 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:22.832111 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:22.833845 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:22.835025 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:22.837308 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:22.838009 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:22.838172 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:22.842258 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:22.842268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:22.842276 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:22.842333 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:22.846439 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:22.847630 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:22.847639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:22.847647 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:22.847857 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:22.848405 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:22.848414 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:22.848418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:22.848874 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:22.849062 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:22.849941 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 10:51:22.850241 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":318,"threads_started":1}
I20260928 10:51:22.851990 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:22.828676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.129:35389)
0928 10:51:22.829609 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:22.829658 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:22.831497 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:22.832387 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:22.832405 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:22.832967 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:22.833626 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:22.833646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:22.835308 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:22.835318 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:22.837057 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:22.837077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:22.838280 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:22.838286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:22.838582 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:22.839430 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:22.839458 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:22.841894 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:22.846836 (+ 4942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:22.846850 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:22.846867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:22.847426 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:22.847994 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:22.847999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:22.848003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:22.848259 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:22.849062 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:22.849069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:22.849463 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 10:51:22.849758 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 10:51:22.850271 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":323,"threads_started":1}
I20260928 10:51:22.853890 12376 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36225
I20260928 10:51:22.854302 12376 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:22.855608 12376 heartbeater.cc:511] Master 127.5.227.190:36225 requested a full tablet report, sending...
I20260928 10:51:22.858462 12187 ts_manager.cc:194] Registered new tserver with Master: 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621)
I20260928 10:51:22.860127 12187 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:35389
WARNING: no policy specified for oryx/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab.
Entry for principal oryx/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab.
I20260928 10:51:22.898106 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36225
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab
--principal=oryx/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:23.267093 12382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:23.267482 12382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:23.267652 12382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:23.275812 12382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:23.275995 12382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:23.276172 12382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:23.287914 12382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab
--principal=oryx/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:23.289340 12382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:23.291070 12382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:23.307525 12390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:23.307689 12387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:23.307629 12388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:23.308082 12382 server_base.cc:1034] running on GCE node
I20260928 10:51:23.308956 12382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:23.310223 12382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:23.311558 12382 hybrid_clock.cc:648] HybridClock initialized: now 1790592683311497 us; error 52 us; skew 500 ppm
I20260928 10:51:23.312068 12382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:23 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:23.315992 12382 init.cc:377] Logged in from keytab as oryx/127.5.227.130@KRBTEST.COM (short username oryx)
I20260928 10:51:23.318353 12382 webserver.cc:460] Webserver started at http://127.5.227.130:42695/ using document root <none> and password file <none>
I20260928 10:51:23.319468 12382 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:23.319660 12382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:23.320046 12382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:23.323841 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "5b7f2fa7b49884d32477ac0492e03b5a"
server_key_iv: "9b52c2de2ceb014929357f0cf1e01dd2"
server_key_version: "encryptionkey@0"
I20260928 10:51:23.324769 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "5b7f2fa7b49884d32477ac0492e03b5a"
server_key_iv: "9b52c2de2ceb014929357f0cf1e01dd2"
server_key_version: "encryptionkey@0"
I20260928 10:51:23.331831 12382 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 10:51:23.336604 12397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:23.338765 12382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:51:23.339075 12382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "5b7f2fa7b49884d32477ac0492e03b5a"
server_key_iv: "9b52c2de2ceb014929357f0cf1e01dd2"
server_key_version: "encryptionkey@0"
I20260928 10:51:23.339344 12382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:23.374884 12382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:23.379805 12382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:23.380111 12382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:23.381161 12382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:23.382722 12382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:23.382830 12382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:23.382990 12382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:23.383061 12382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:23.420126 12382 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:45421
I20260928 10:51:23.420178 12509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:45421 every 8 connection(s)
I20260928 10:51:23.421777 12382 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:23.423324 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 12382
I20260928 10:51:23.423621 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:51:23.424355 6030 external_mini_cluster.cc:1428] Setting key 7155058d9eb2aef90e5d862eb8ca1170
Sep 28 10:51:23 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:51:23.449226 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:23.425238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:34451 (local address 127.5.227.190:36225)
0928 10:51:23.425741 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:23.425745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:23.427285 (+ 1540us) server_negotiation.cc:407] Connection header received
0928 10:51:23.429522 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:23.429531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:23.429666 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:23.429788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:23.432246 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:23.433225 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:23.436115 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:23.436591 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:23.436731 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:23.442311 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:23.442317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:23.442320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:23.442359 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:23.445750 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:23.446637 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:23.446644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:23.446647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:23.446785 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:23.447259 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:23.447264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:23.447267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:23.447573 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:23.447766 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:23.448555 (+ 789us) server_negotiation.cc:299] Negotiation successful
0928 10:51:23.448837 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":283,"threads_started":1}
I20260928 10:51:23.450445 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:23.426069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.130:34451)
0928 10:51:23.427087 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:23.427149 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:23.429184 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:23.430252 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:23.430263 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:23.430997 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:23.431992 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:23.432021 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:23.433434 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:23.433439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:23.435812 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:23.435839 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:23.436914 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:23.436924 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:23.437312 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:23.438273 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:23.438315 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:23.441954 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:23.445943 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:23.445953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:23.445988 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:23.446482 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:23.446902 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:23.446907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:23.446911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:23.447127 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:23.447777 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:23.447787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:23.448174 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 10:51:23.448456 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 10:51:23.449062 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":448,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.131@KRBTEST.COM" created.
I20260928 10:51:23.452338 12510 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36225
I20260928 10:51:23.452667 12510 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:23.453485 12510 heartbeater.cc:511] Master 127.5.227.190:36225 requested a full tablet report, sending...
I20260928 10:51:23.455402 12187 ts_manager.cc:194] Registered new tserver with Master: fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421)
I20260928 10:51:23.456929 12187 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:34451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab.
Entry for principal oryx/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab.
I20260928 10:51:23.487799 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36225
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab
--principal=oryx/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:23.852699 12516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:23.853044 12516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:23.853202 12516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:23.861771 12516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:23.861923 12516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:23.862214 12516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:23.863890 12376 heartbeater.cc:503] Master 127.5.227.190:36225 was elected leader, sending a full tablet report...
I20260928 10:51:23.874455 12516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab
--principal=oryx/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:23.876107 12516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:23.878049 12516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:23.895617 12524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:23.895815 12521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:23.895939 12522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:23.896441 12516 server_base.cc:1034] running on GCE node
I20260928 10:51:23.897414 12516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:23.898999 12516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:23.900296 12516 hybrid_clock.cc:648] HybridClock initialized: now 1790592683900248 us; error 78 us; skew 500 ppm
I20260928 10:51:23.900935 12516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:23 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:23.905462 12516 init.cc:377] Logged in from keytab as oryx/127.5.227.131@KRBTEST.COM (short username oryx)
I20260928 10:51:23.907694 12516 webserver.cc:460] Webserver started at http://127.5.227.131:46545/ using document root <none> and password file <none>
I20260928 10:51:23.908912 12516 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:23.909180 12516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:23.909530 12516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:23.913224 12516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "6458de1cef7a42e6b3344e806939474d"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "02c9f4b6e5992890a4d54c05d07f31e4"
server_key_iv: "9468db3624ff1618c89a771d2d0175d4"
server_key_version: "encryptionkey@0"
I20260928 10:51:23.914449 12516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "6458de1cef7a42e6b3344e806939474d"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "02c9f4b6e5992890a4d54c05d07f31e4"
server_key_iv: "9468db3624ff1618c89a771d2d0175d4"
server_key_version: "encryptionkey@0"
I20260928 10:51:23.922291 12516 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 10:51:23.928125 12531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:23.930298 12516 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:51:23.930552 12516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "6458de1cef7a42e6b3344e806939474d"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "02c9f4b6e5992890a4d54c05d07f31e4"
server_key_iv: "9468db3624ff1618c89a771d2d0175d4"
server_key_version: "encryptionkey@0"
I20260928 10:51:23.930805 12516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:23.963711 12516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:23.970237 12516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:23.970837 12516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:23.972206 12516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:23.973793 12516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:23.973876 12516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:23.973966 12516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:23.974012 12516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:24.020745 12516 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:33795
I20260928 10:51:24.020807 12643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:33795 every 8 connection(s)
I20260928 10:51:24.022600 12516 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:24.024027 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 12516
I20260928 10:51:24.024278 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:51:24.024842 6030 external_mini_cluster.cc:1428] Setting key 28e3de9ccfb302ba8eff662ffa551bce
Sep 28 10:51:24 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:24.051127 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.027041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:60447 (local address 127.5.227.190:36225)
0928 10:51:24.027834 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.027841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.029323 (+ 1482us) server_negotiation.cc:407] Connection header received
0928 10:51:24.031609 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.031614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.031757 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.031910 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.033825 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.034822 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.037063 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.037484 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.037650 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.042780 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.042787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.042792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.042840 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.046711 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.048114 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.048119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.048122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.048280 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.048968 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.048973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.048975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.049377 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.049636 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.050569 (+ 933us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.050773 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":488,"threads_started":1}
I20260928 10:51:24.052242 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.027757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.131:60447)
0928 10:51:24.029105 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:24.029151 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:24.031250 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:24.032310 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:24.032320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:24.032938 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:24.033591 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.033613 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.035013 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.035021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.036847 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.036863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.037755 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.037765 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.038039 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.038896 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:24.038923 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:24.042476 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:24.046934 (+ 4458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.046943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.047018 (+ 75us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.047945 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.048522 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.048527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.048529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.048781 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.049679 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:24.049687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:24.050099 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 10:51:24.050383 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 10:51:24.050827 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":459,"threads_started":1}
I20260928 10:51:24.054750 12644 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36225
I20260928 10:51:24.055233 12644 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:24.056322 12644 heartbeater.cc:511] Master 127.5.227.190:36225 requested a full tablet report, sending...
I20260928 10:51:24.058777 12187 ts_manager.cc:194] Registered new tserver with Master: 6458de1cef7a42e6b3344e806939474d (127.5.227.131:33795)
I20260928 10:51:24.059788 12187 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:60447
I20260928 10:51:24.069689 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.227.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.189.keytab.
Entry for principal oryx/127.5.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.189.keytab.
I20260928 10:51:24.147921 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:33863
--webserver_interface=127.5.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.189.keytab
--principal=oryx/127.5.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:36225,127.5.227.189:33863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:24.460605 12510 heartbeater.cc:503] Master 127.5.227.190:36225 was elected leader, sending a full tablet report...
W20260928 10:51:24.525146 12650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:24.525522 12650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:24.525700 12650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:24.533725 12650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:24.533923 12650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:24.534013 12650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:24.534118 12650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:24.534185 12650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:24.544840 12650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:36225,127.5.227.189:33863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.189.keytab
--principal=oryx/127.5.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:33863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
W20260928 10:51:24.545298 12650 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.227.190:36225,127.5.227.189:33863'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 10:51:24.546393 12650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:24.547816 12650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:24.564121 12658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:24.564119 12655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:24.564183 12656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:24.564971 12650 server_base.cc:1034] running on GCE node
I20260928 10:51:24.565863 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:24.567539 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:24.568871 12650 hybrid_clock.cc:648] HybridClock initialized: now 1790592684568766 us; error 76 us; skew 500 ppm
I20260928 10:51:24.569289 12650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:24 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592684, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:24.573172 12650 init.cc:377] Logged in from keytab as oryx/127.5.227.189@KRBTEST.COM (short username oryx)
I20260928 10:51:24.575181 12650 webserver.cc:460] Webserver started at http://127.5.227.189:40975/ using document root <none> and password file <none>
I20260928 10:51:24.576072 12650 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:24.576218 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:24.576598 12650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:24.580488 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/instance:
uuid: "d827d2628e984b3195cea1e7a4786634"
format_stamp: "Formatted at 2026-09-28 10:51:24 on dist-test-slave-8w8g"
server_key: "fea3aee48c3223a763ac0190807ecdfd"
server_key_iv: "10941e9b39ed2be62cee00a98b7035ef"
server_key_version: "encryptionkey@0"
I20260928 10:51:24.581537 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal/instance:
uuid: "d827d2628e984b3195cea1e7a4786634"
format_stamp: "Formatted at 2026-09-28 10:51:24 on dist-test-slave-8w8g"
server_key: "fea3aee48c3223a763ac0190807ecdfd"
server_key_iv: "10941e9b39ed2be62cee00a98b7035ef"
server_key_version: "encryptionkey@0"
I20260928 10:51:24.588079 12650 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:51:24.592974 12665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:24.595014 12650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:51:24.595216 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
uuid: "d827d2628e984b3195cea1e7a4786634"
format_stamp: "Formatted at 2026-09-28 10:51:24 on dist-test-slave-8w8g"
server_key: "fea3aee48c3223a763ac0190807ecdfd"
server_key_iv: "10941e9b39ed2be62cee00a98b7035ef"
server_key_version: "encryptionkey@0"
I20260928 10:51:24.595469 12650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:24.629192 12650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:24.634928 12650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:24.635274 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:24.658752 12650 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.189:33863
I20260928 10:51:24.658802 12716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.189:33863 every 8 connection(s)
I20260928 10:51:24.660326 12650 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
I20260928 10:51:24.664362 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 12650
I20260928 10:51:24.665110 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:24.676200 12717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:51:24 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.189@KRBTEST.COM
Sep 28 10:51:24 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592684, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.189@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:24.697652 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.666974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.5.227.189:33863)
0928 10:51:24.667882 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.667894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.667972 (+ 78us) server_negotiation.cc:407] Connection header received
0928 10:51:24.670507 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.670537 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.671055 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.671474 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.673769 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.675404 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.677701 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.678768 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.679023 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.685856 (+ 6833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.685867 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.685892 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.686024 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.690594 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.692234 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.692243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.692251 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.692556 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.693131 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.693136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.693138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.694064 (+ 926us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.694439 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.695225 (+ 786us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.695837 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"spinlock_wait_cycles":20864,"thread_start_us":378,"threads_started":1}
I20260928 10:51:24.701783 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.680785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51330)
0928 10:51:24.681775 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:24.681847 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:24.682115 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:24.683096 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:24.683107 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:24.683984 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:24.684258 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.684268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.685784 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.685791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.687819 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.687848 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.689039 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.689047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.689266 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.689999 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:24.690029 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:24.693367 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:24.698435 (+ 5068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.698446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.698452 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.699007 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.699511 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.699516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.699520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.699762 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.700563 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:24.700570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:24.700763 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 10:51:24.700996 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 10:51:24.701413 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":419,"threads_started":1}
I20260928 10:51:24.701812 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.680859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.5.227.190:36225)
0928 10:51:24.682673 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.682678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.682696 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:51:24.682764 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.682768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.682834 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.682939 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.684471 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.685599 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.688156 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.688787 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.688914 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.693639 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.693647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.693651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.693714 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.698193 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.699193 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.699200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.699204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.699388 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.699916 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.699920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.699922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.700346 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.700597 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.701223 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.701544 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1690,"thread_start_us":1587,"threads_started":1}
I20260928 10:51:24.703670 12717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:51:24 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592684, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.189@KRBTEST.COM for oryx/127.5.227.189@KRBTEST.COM
I20260928 10:51:24.719837 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.704753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.0.0.1:51954)
0928 10:51:24.705022 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:24.705043 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:24.705247 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:24.705562 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:24.705567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:24.705988 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:24.706138 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.706147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.707301 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.707307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.708350 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.708365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.709146 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.709152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.709402 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.710187 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:24.710208 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:24.712936 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:24.717063 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.717068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.717071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.717481 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.717891 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.717899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.717903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.718041 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.718647 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:24.718652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:24.718790 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 10:51:24.718929 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 10:51:24.719261 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":1}
I20260928 10:51:24.719938 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.704786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.5.227.189:33863)
0928 10:51:24.705017 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.705024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.705164 (+ 140us) server_negotiation.cc:407] Connection header received
0928 10:51:24.705367 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.705372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.705438 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.705547 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.706283 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.707117 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.708512 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.708972 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.709133 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.713356 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.713362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.713365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.713405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.716857 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.717628 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.717636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.717641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.717779 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.718207 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.718212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.718215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.718520 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.718689 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.719003 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.719423 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":233,"server-negotiator.queue_time_us":72}
I20260928 10:51:24.727276 12717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Bootstrap starting.
I20260928 10:51:24.731115 12717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:24.732375 12717 log.cc:824] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:24.736064 12717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: No bootstrap required, opened a new log
I20260928 10:51:24.745339 12717 raft_consensus.cc:348] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } }
I20260928 10:51:24.746064 12717 raft_consensus.cc:374] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:24.746209 12717 raft_consensus.cc:729] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d827d2628e984b3195cea1e7a4786634, State: Initialized, Role: FOLLOWER
I20260928 10:51:24.747071 12717 consensus_queue.cc:260] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } }
I20260928 10:51:24.748198 12724 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } } }
I20260928 10:51:24.748515 12724 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:51:24.749436 12717 sys_catalog.cc:564] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 10:51:24.761842 12735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:51:24.762036 12735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:51:24.763213 12650 master_runner.cc:428] Detected that this master d827d2628e984b3195cea1e7a4786634 is joining an existing cluster
I20260928 10:51:24.763315 12650 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.227.189:33863
I20260928 10:51:24.784873 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.767768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.0.0.1:51962)
0928 10:51:24.768796 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:24.768818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:24.769004 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:24.769421 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:24.769424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:24.770032 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:24.770455 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.770473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.772114 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.772117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.773328 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.773342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.774297 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.774304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.774504 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.775241 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:24.775269 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:24.776592 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:24.781234 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.781243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.781248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.781788 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.782418 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.782425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.782437 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.782778 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.783481 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:24.783485 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:24.783645 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 10:51:24.783878 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 10:51:24.784226 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":24,"thread_start_us":701,"threads_started":1}
I20260928 10:51:24.785496 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.767441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51338)
0928 10:51:24.768529 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:24.768550 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:24.768792 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:24.769192 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:24.769197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:24.769967 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:24.770489 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.770502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.772037 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.772043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.774126 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.774142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.775434 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.775443 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.775660 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.776429 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:24.776454 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:24.777374 (+ 920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:24.780904 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.780910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.780915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.781375 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.782444 (+ 1069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.782448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.782451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.782778 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.784185 (+ 1407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:24.784190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:24.784377 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 10:51:24.784659 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 10:51:24.785052 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":59,"thread_start_us":567,"threads_started":1}
I20260928 10:51:24.785857 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.767950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.5.227.189:33863)
0928 10:51:24.768200 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.768210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.768910 (+ 700us) server_negotiation.cc:407] Connection header received
0928 10:51:24.769139 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.769144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.769203 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.769296 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.770620 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.771852 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.773507 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.774097 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.774251 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.776859 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.776868 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.776873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.776915 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.781079 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.782185 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.782189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.782191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.782305 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.782919 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.782925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.782929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.783335 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.783503 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.783971 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.785312 (+ 1341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1173,"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":44800}
I20260928 10:51:24.786120 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.767357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.5.227.190:36225)
0928 10:51:24.767564 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.767570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.768672 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 10:51:24.768938 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.768944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.769031 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.769147 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.770717 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.771852 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.774341 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.775055 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.775214 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.777608 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.777630 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.777640 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.777687 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.780729 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.781587 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.781594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.781597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.781766 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.783578 (+ 1812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.783595 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.783599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.784036 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.784200 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.784785 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.785472 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 10:51:24 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.190@KRBTEST.COM for oryx/127.5.227.189@KRBTEST.COM
I20260928 10:51:24.810797 12724 raft_consensus.cc:482] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:24.811225 12724 raft_consensus.cc:504] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } }
I20260928 10:51:24.812780 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.792793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.0.0.1:51972)
0928 10:51:24.793657 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:24.793700 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:24.793916 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:24.794493 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:24.794499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:24.795269 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:24.795523 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.795534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.797051 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.797056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.798198 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.798212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.799390 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.799395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.799661 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.800730 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:24.800802 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:24.803267 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:24.808499 (+ 5232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.808521 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.808533 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.809249 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.809866 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.809873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.809880 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.810098 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.811384 (+ 1286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:24.811397 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:24.811637 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 10:51:24.811914 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 10:51:24.812311 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":419,"threads_started":1}
I20260928 10:51:24.813103 12724 leader_election.cc:288] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225)
I20260928 10:51:24.813800 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.792836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.5.227.189:33863)
0928 10:51:24.793041 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.793049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.793814 (+ 765us) server_negotiation.cc:407] Connection header received
0928 10:51:24.794185 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.794191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.794307 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.794440 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.795702 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.796845 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.798389 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.799143 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.799300 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.803543 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.803551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.803556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.803602 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.808236 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.809420 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.809426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.809430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.809706 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.810566 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.810572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.810575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.811114 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.811278 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.813165 (+ 1887us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.813420 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:51:24.814523 12197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d827d2628e984b3195cea1e7a4786634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaae6cf9d0245659e76c2d34361d683" is_pre_election: true
I20260928 10:51:24.816185 12187 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.227.189:33863: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d827d2628e984b3195cea1e7a4786634"
member_type: NON_VOTER
last_known_addr {
host: "127.5.227.189"
port: 33863
}
attrs {
promote: true
}
}
dest_uuid: "bbaae6cf9d0245659e76c2d34361d683"
cas_config_opid_index: -1
I20260928 10:51:24.816228 12666 leader_election.cc:302] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: d827d2628e984b3195cea1e7a4786634; no voters: bbaae6cf9d0245659e76c2d34361d683
I20260928 10:51:24.817167 12724 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 10:51:24.817641 12187 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } }
I20260928 10:51:24.829815 12692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.5.227.188@KRBTEST.COM; defaulting to no policy
I20260928 10:51:24.832142 12747 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d827d2628e984b3195cea1e7a4786634 (127.5.227.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:24.833871 12692 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Refusing update from remote peer bbaae6cf9d0245659e76c2d34361d683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 10:51:24.835253 12746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.188@KRBTEST.COM" created.
I20260928 10:51:24.859797 12747 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.227.189:33863
I20260928 10:51:24.860544 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:24.860874 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:24.860894 12650 master.cc:537] Master@127.5.227.189:33863 shutting down...
I20260928 10:51:24.876689 12692 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d827d2628e984b3195cea1e7a4786634 (127.5.227.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:24.883667 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:24.884613 12752 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:24.884948 12752 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:51:24.884896 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:51:24.884894 12724 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbaae6cf9d0245659e76c2d34361d683. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:24.885150 12724 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: LEARNER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.188.keytab.
Entry for principal oryx/127.5.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.188.keytab.
I20260928 10:51:24.897883 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:34513
--webserver_interface=127.5.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.188.keytab
--principal=oryx/127.5.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:24.914757 12650 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 LEARNER]: Raft consensus shutting down.
I20260928 10:51:24.915314 12650 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 LEARNER]: Raft consensus is shut down!
I20260928 10:51:24.915508 12650 tablet_replica.cc:330] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: stopping tablet replica
I20260928 10:51:24.935401 12650 master.cc:552] Master@127.5.227.189:33863 shutdown complete.
I20260928 10:51:24.935524 12650 master_runner.cc:305] Clearing existing system tablet
I20260928 10:51:24.936079 12650 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:24.936203 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:24.939260 12650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 10:51:24.941171 12760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:24.941634 12650 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:24.941962 12650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 10:51:24.942062 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
uuid: "d827d2628e984b3195cea1e7a4786634"
format_stamp: "Formatted at 2026-09-28 10:51:24 on dist-test-slave-8w8g"
server_key: "fea3aee48c3223a763ac0190807ecdfd"
server_key_iv: "10941e9b39ed2be62cee00a98b7035ef"
server_key_version: "encryptionkey@0"
I20260928 10:51:24.946643 12650 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:24.953562 12650 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 10:51:24.953830 12650 log.cc:1196] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 10:51:24.954353 12650 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Deleting consensus metadata
I20260928 10:51:24.955089 12650 master_runner.cc:315] Copying system tablet from 127.5.227.190:36225
I20260928 10:51:24.961974 12650 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: tablet copy: Beginning tablet copy session from remote peer at address 127.5.227.190:36225
I20260928 10:51:24.977444 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.962907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.5.227.190:36225)
0928 10:51:24.963241 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:24.963252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:24.963990 (+ 738us) server_negotiation.cc:407] Connection header received
0928 10:51:24.964288 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:24.964299 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:24.964408 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:24.964525 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:24.965523 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.966337 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.968098 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.968801 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.968962 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.971147 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:24.971152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:24.971156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:24.971193 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:24.974218 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.975051 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.975057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.975061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.975235 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:24.975709 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:24.975714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:24.975716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:24.976102 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:24.976278 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:24.976751 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 10:51:24.977018 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 10:51:24.977535 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:24.962887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51350)
0928 10:51:24.963846 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:24.963866 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:24.964116 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:24.964557 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:24.964562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:24.965089 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:24.965340 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.965351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.966519 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.966526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.967870 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:24.967898 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:24.969004 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:24.969012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:24.969240 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:24.970002 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:24.970022 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:24.970890 (+ 868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:24.974406 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.974412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.974417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.974869 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.975379 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:24.975385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:24.975389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:24.975551 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:24.976242 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:24.976248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:24.976471 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 10:51:24.976646 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 10:51:24.977054 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":596,"threads_started":1}
I20260928 10:51:24.978861 12207 tablet_copy_service.cc:140] P bbaae6cf9d0245659e76c2d34361d683: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d827d2628e984b3195cea1e7a4786634 ({username='oryx', principal='oryx/127.5.227.189@KRBTEST.COM'} at 127.0.0.1:51350)
I20260928 10:51:24.979151 12207 tablet_copy_service.cc:161] P bbaae6cf9d0245659e76c2d34361d683: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d827d2628e984b3195cea1e7a4786634 at {username='oryx', principal='oryx/127.5.227.189@KRBTEST.COM'} at 127.0.0.1:51350: session id = d827d2628e984b3195cea1e7a4786634-00000000000000000000000000000000
I20260928 10:51:24.982914 12207 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 10:51:24.985239 12650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 10:51:24.991279 12650 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: tablet copy: Starting download of 0 data blocks...
I20260928 10:51:24.991899 12650 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: tablet copy: Starting download of 1 WAL segments...
I20260928 10:51:24.994791 12650 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 10:51:24.999488 12207 tablet_copy_service.cc:342] P bbaae6cf9d0245659e76c2d34361d683: Request end of tablet copy session d827d2628e984b3195cea1e7a4786634-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.227.189@KRBTEST.COM'} at 127.0.0.1:51350
I20260928 10:51:24.999727 12207 tablet_copy_service.cc:433] P bbaae6cf9d0245659e76c2d34361d683: ending tablet copy session d827d2628e984b3195cea1e7a4786634-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d827d2628e984b3195cea1e7a4786634
I20260928 10:51:25.015841 12650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:25.028192 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:25.031638 12768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:25.032338 12650 server_base.cc:1034] running on GCE node
W20260928 10:51:25.032702 12771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:25.033310 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:25.034067 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:25.035311 12650 hybrid_clock.cc:648] HybridClock initialized: now 1790592685035217 us; error 64 us; skew 500 ppm
I20260928 10:51:25.035590 12650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592685, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:25.038688 12650 init.cc:377] Logged in from keytab as oryx/127.5.227.189@KRBTEST.COM (short username oryx)
I20260928 10:51:25.040798 12650 webserver.cc:460] Webserver started at http://127.5.227.189:33421/ using document root <none> and password file <none>
I20260928 10:51:25.041218 12650 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:25.041324 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:25.045219 12650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:51:25.048228 12777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:25.048962 12650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 10:51:25.049139 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
uuid: "d827d2628e984b3195cea1e7a4786634"
format_stamp: "Formatted at 2026-09-28 10:51:24 on dist-test-slave-8w8g"
server_key: "fea3aee48c3223a763ac0190807ecdfd"
server_key_iv: "10941e9b39ed2be62cee00a98b7035ef"
server_key_version: "encryptionkey@0"
I20260928 10:51:25.049327 12650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:25.063844 12644 heartbeater.cc:503] Master 127.5.227.190:36225 was elected leader, sending a full tablet report...
I20260928 10:51:25.069002 12650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:25.075965 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:25.168295 12650 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.189:33863
I20260928 10:51:25.168327 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.189:33863 every 8 connection(s)
I20260928 10:51:25.169987 12650 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
I20260928 10:51:25.179154 12829 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:51:25.185000 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Bootstrap starting.
I20260928 10:51:25.201435 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:51:25.202015 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: Bootstrap complete.
I20260928 10:51:25.203030 12829 raft_consensus.cc:348] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } }
I20260928 10:51:25.203274 12829 raft_consensus.cc:729] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d827d2628e984b3195cea1e7a4786634, State: Initialized, Role: LEARNER
I20260928 10:51:25.203528 12829 consensus_queue.cc:260] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } }
I20260928 10:51:25.204443 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:25.204881 12829 sys_catalog.cc:564] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:25.204882 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:51:25.216681 12843 catalog_manager.cc:1259] Loaded cluster ID: ecb34afe09b145fcb8491764a29d4a70
I20260928 10:51:25.216815 12843 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: loading cluster ID for follower catalog manager: success
I20260928 10:51:25.221635 12843 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: acquiring CA information for follower catalog manager: success
I20260928 10:51:25.225261 12843 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:51:25.380411 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:25.380846 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:25.399480 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.380681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.0.0.1:51988)
0928 10:51:25.393280 (+ 12599us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.393306 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.393588 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.394088 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.394093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.394124 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:25.394325 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.394337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.395660 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.395666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.397212 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.397229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.398549 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.398555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.398756 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.398824 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.398951 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.399105 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12461,"thread_start_us":12342,"threads_started":1}
I20260928 10:51:25.399466 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.381134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.5.227.189:33863)
0928 10:51:25.389642 (+ 8508us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.389654 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.393431 (+ 3777us) server_negotiation.cc:407] Connection header received
0928 10:51:25.393810 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.393821 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.393901 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.394040 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:25.394487 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.395479 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.397409 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.398320 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.398481 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.398764 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.398986 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.399126 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8170,"thread_start_us":7953,"threads_started":1}
I20260928 10:51:25.400786 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbaae6cf9d0245659e76c2d34361d683. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:25.401104 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:51:25.401822 12845 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: attempting to promote NON_VOTER d827d2628e984b3195cea1e7a4786634 to VOTER
W20260928 10:51:25.402288 12758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:25.402635 12758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:25.402789 12758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 10:51:25.402675 12845 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } }
I20260928 10:51:25.405045 12804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 LEARNER]: Refusing update from remote peer bbaae6cf9d0245659e76c2d34361d683: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 10:51:25.405828 12844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 10:51:25.410218 12845 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d827d2628e984b3195cea1e7a4786634 (127.5.227.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } }
I20260928 10:51:25.411033 12804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d827d2628e984b3195cea1e7a4786634 (127.5.227.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } }
W20260928 10:51:25.413357 12758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:25.413511 12758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:25.413601 12758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:25.413687 12758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:25.413754 12758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:25.414765 12845 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } }
I20260928 10:51:25.415082 12845 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:25.415004 12844 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } }
I20260928 10:51:25.415261 12844 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:25.415591 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } }
I20260928 10:51:25.415848 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:51:25.426219 12758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.188.keytab
--principal=oryx/127.5.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:25.427690 12758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:25.429395 12758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:25.443243 12854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:25.443379 12855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:25.443325 12857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:25.444631 12758 server_base.cc:1034] running on GCE node
I20260928 10:51:25.445662 12758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:25.447330 12758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:25.449074 12758 hybrid_clock.cc:648] HybridClock initialized: now 1790592685448543 us; error 92 us; skew 500 ppm
I20260928 10:51:25.449528 12758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592685, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:25.453167 12758 init.cc:377] Logged in from keytab as oryx/127.5.227.188@KRBTEST.COM (short username oryx)
I20260928 10:51:25.454869 12758 webserver.cc:460] Webserver started at http://127.5.227.188:38731/ using document root <none> and password file <none>
I20260928 10:51:25.455605 12758 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:25.455715 12758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:25.456142 12758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:25.459887 12758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/instance:
uuid: "93894c132dae473d838d5f9bf5b3fba4"
format_stamp: "Formatted at 2026-09-28 10:51:25 on dist-test-slave-8w8g"
server_key: "5f41eb1cfa0b096c79dc57bf94e995e1"
server_key_iv: "b33961fac96ca83b17efaf446a99f1b9"
server_key_version: "encryptionkey@0"
I20260928 10:51:25.460785 12758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal/instance:
uuid: "93894c132dae473d838d5f9bf5b3fba4"
format_stamp: "Formatted at 2026-09-28 10:51:25 on dist-test-slave-8w8g"
server_key: "5f41eb1cfa0b096c79dc57bf94e995e1"
server_key_iv: "b33961fac96ca83b17efaf446a99f1b9"
server_key_version: "encryptionkey@0"
I20260928 10:51:25.467403 12758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260928 10:51:25.471688 12864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:25.473500 12758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:51:25.473706 12758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
uuid: "93894c132dae473d838d5f9bf5b3fba4"
format_stamp: "Formatted at 2026-09-28 10:51:25 on dist-test-slave-8w8g"
server_key: "5f41eb1cfa0b096c79dc57bf94e995e1"
server_key_iv: "b33961fac96ca83b17efaf446a99f1b9"
server_key_version: "encryptionkey@0"
I20260928 10:51:25.473896 12758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:25.499833 12758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:25.504885 12758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:25.505215 12758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:25.528218 12758 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.188:34513
I20260928 10:51:25.528400 12915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.188:34513 every 8 connection(s)
I20260928 10:51:25.529886 12758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
I20260928 10:51:25.530561 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 12758
I20260928 10:51:25.535707 12916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:25.544971 12916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.188@KRBTEST.COM
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592685, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.188@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:25.560395 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.532944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.5.227.188:34513)
0928 10:51:25.533952 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.533970 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.534310 (+ 340us) server_negotiation.cc:407] Connection header received
0928 10:51:25.536539 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.536586 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.537051 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.537515 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.539111 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.541122 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.542505 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.543299 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.543536 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.549872 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.549886 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.549909 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.549978 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.554200 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.555199 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.555209 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.555220 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.555464 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.556053 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.556058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.556061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.556855 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.557165 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.558477 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.558960 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":470,"threads_started":1}
I20260928 10:51:25.568991 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.548703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51360)
0928 10:51:25.550310 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.550363 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.550541 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.551148 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.551157 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.552320 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.552748 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.552764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.553929 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.553938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.555402 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.555415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.556541 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.556548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.556714 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.557636 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.557660 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.561233 (+ 3573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.565404 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.565414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.565420 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.566261 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.566870 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.566874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.566877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.567086 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.567854 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.567861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.568111 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.568296 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.568599 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"thread_start_us":1071,"threads_started":1}
I20260928 10:51:25.569248 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.548828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.5.227.190:36225)
0928 10:51:25.549401 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.549407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.550467 (+ 1060us) server_negotiation.cc:407] Connection header received
0928 10:51:25.550682 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.550686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.550788 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.551111 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.552984 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.553767 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.555609 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.556323 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.556442 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.561562 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.561580 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.561585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.561627 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.565216 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.566524 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.566530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.566532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.566719 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.567216 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.567220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.567222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.567624 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.567811 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.568583 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.568811 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":290,"threads_started":1}
I20260928 10:51:25.571817 12916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592685, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.188@KRBTEST.COM for oryx/127.5.227.189@KRBTEST.COM
I20260928 10:51:25.595554 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.572850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.0.0.1:51990)
0928 10:51:25.573162 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.573181 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.573299 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.580758 (+ 7459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.580764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.581255 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.581487 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.581505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.582907 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.582915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.584028 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.584048 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.584890 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.584894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.585045 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.585748 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.585767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.588827 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.592722 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.592727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.592737 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.593211 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.593823 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.593828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.593830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.594038 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.594696 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.594700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.594854 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.594995 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.595266 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260928 10:51:25.595657 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.579978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.5.227.189:33863)
0928 10:51:25.580176 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.580183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.580208 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:51:25.580285 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.580290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.580376 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.580520 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.581703 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.582746 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.584256 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.584720 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.584834 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.589146 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.589152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.589156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.589204 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.592501 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.593366 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.593371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.593373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.593667 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.594171 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.594175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.594177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.594530 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.594652 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.595134 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.595351 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:51:25.596680 12916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592685, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.188@KRBTEST.COM for oryx/127.5.227.188@KRBTEST.COM
I20260928 10:51:25.610430 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.597415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.0.0.1:40514)
0928 10:51:25.597633 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.597649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.597786 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.598113 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.598117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.598527 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.598668 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.598676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.599616 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.599623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.600570 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.600582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.601271 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.601277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.601452 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.602045 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.602061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.604390 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.607791 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.607805 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.607810 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.608217 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.608637 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.608642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.608645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.608791 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.609340 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.609344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.609479 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.609612 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.609967 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 10:51:25.610514 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.597626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.5.227.188:34513)
0928 10:51:25.597786 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.597792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.597808 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:51:25.597893 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.597899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.597998 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.598099 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.598797 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.599490 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.600706 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.601108 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.601230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.604601 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.604606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.604609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.604639 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.607564 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.608355 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.608362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.608380 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.608515 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.608923 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.608930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.608935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.609223 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.609377 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.609692 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.610133 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":206,"server-negotiator.queue_time_us":68}
I20260928 10:51:25.616585 12916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Bootstrap starting.
I20260928 10:51:25.620249 12916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:25.621428 12916 log.cc:824] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:25.624652 12916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: No bootstrap required, opened a new log
I20260928 10:51:25.632438 12916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } }
I20260928 10:51:25.633068 12916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:25.633255 12916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93894c132dae473d838d5f9bf5b3fba4, State: Initialized, Role: FOLLOWER
I20260928 10:51:25.634025 12916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } }
I20260928 10:51:25.635193 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } } }
I20260928 10:51:25.635535 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:51:25.637059 12916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 10:51:25.648082 12934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:51:25.648257 12934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:51:25.654004 12758 master_runner.cc:428] Detected that this master 93894c132dae473d838d5f9bf5b3fba4 is joining an existing cluster
I20260928 10:51:25.654150 12758 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.227.188:34513
I20260928 10:51:25.668960 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 12248
I20260928 10:51:25.679654 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.658975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51372)
0928 10:51:25.660594 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.660613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.660813 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.661496 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.661499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.662015 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.662313 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.662323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.663891 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.663898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.665674 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.665689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.667624 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.667639 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.667927 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.668925 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.668957 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.670224 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.675022 (+ 4798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.675028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.675032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.675976 (+ 944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.676902 (+ 926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.676909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.676913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.677177 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.678175 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.678180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.678416 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.678600 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.678977 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"thread_start_us":836,"threads_started":1}
I20260928 10:51:25.680843 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.660022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.5.227.190:36225)
0928 10:51:25.660256 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.660264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.660729 (+ 465us) server_negotiation.cc:407] Connection header received
0928 10:51:25.661065 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.661072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.661170 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.661260 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.662491 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.663641 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.665851 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.666680 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.666807 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.670479 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.670490 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.670497 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.670547 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.674829 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.676145 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.676152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.676156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.676345 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.677343 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.677350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.677362 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.677917 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.678102 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.680089 (+ 1987us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.680352 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:51:25.683498 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.659589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.5.227.189:33863)
0928 10:51:25.659824 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.659834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.663114 (+ 3280us) server_negotiation.cc:407] Connection header received
0928 10:51:25.663341 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.663348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.663433 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.663499 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.665119 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.666194 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.668692 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.669945 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.670060 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.674265 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.674276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.674284 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.674343 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.678992 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.680144 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.680153 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.680159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.680361 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.681069 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.681075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.681078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.681572 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.681760 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.682612 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.682987 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:51:25.685197 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.659487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.0.0.1:51992)
0928 10:51:25.662991 (+ 3504us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.663025 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.663210 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.663704 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.663711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.664555 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.664912 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.664925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.666409 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.666415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.668510 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.668525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.670680 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.670692 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.670878 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.672247 (+ 1369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.672282 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.673937 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.679286 (+ 5349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.679294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.679301 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.679979 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.680657 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.680665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.680672 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.680901 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.681876 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.681884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.682196 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.683824 (+ 1628us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.684387 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3361,"thread_start_us":10005,"threads_started":1}
I20260928 10:51:25.693308 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.661157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.5.227.188:34513)
0928 10:51:25.661346 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.661351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.669506 (+ 8155us) server_negotiation.cc:407] Connection header received
0928 10:51:25.669964 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.669969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.670042 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.670112 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.672646 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.673924 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.675997 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.676834 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.677034 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.681400 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.681407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.681410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.681445 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.689240 (+ 7795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.690355 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.690361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.690368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.690557 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.691385 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.691389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.691392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.691858 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.692034 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.692550 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.692813 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:51:25.695111 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.660900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.0.0.1:40522)
0928 10:51:25.669390 (+ 8490us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.669416 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.669813 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.670302 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.670306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.671933 (+ 1627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.672343 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.672363 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.674159 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.674164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.675823 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.675841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.677379 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.677386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.677694 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.679672 (+ 1978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.679699 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.681187 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.689423 (+ 8236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.689431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.689440 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.690097 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.691095 (+ 998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.691100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.691103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.691277 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.692103 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.692109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.692284 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.694218 (+ 1934us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.694563 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8309,"thread_start_us":1723,"threads_started":1}
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.190@KRBTEST.COM for oryx/127.5.227.188@KRBTEST.COM
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.129@KRBTEST.COM
I20260928 10:51:25.712088 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.688131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.0.0.1:40524)
0928 10:51:25.688340 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.688361 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.688668 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.694106 (+ 5438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.694112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.694657 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.694865 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.694878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.696742 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.696748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.697793 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.697810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.698639 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.698644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.698834 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.699637 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.699662 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.703216 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.708886 (+ 5670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.708892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.708896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.709577 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.710159 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.710166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.710171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.710377 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.711017 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.711022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.711203 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.711355 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.711668 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 10:51:25.712109 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.688527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.5.227.188:34513)
0928 10:51:25.693683 (+ 5156us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.693689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.693711 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:51:25.693807 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.693813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.693934 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.694052 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.695087 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.696543 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.697984 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.698460 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.698612 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.703469 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.703477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.703481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.703527 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.708636 (+ 5109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.709815 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.709821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.709825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.710004 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.710530 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.710535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.710539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.710858 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.711039 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.711459 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.711683 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5055,"thread_start_us":15775,"threads_started":1}
I20260928 10:51:25.714188 12187 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.227.188:34513: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4"
member_type: NON_VOTER
last_known_addr {
host: "127.5.227.188"
port: 34513
}
attrs {
promote: true
}
}
dest_uuid: "bbaae6cf9d0245659e76c2d34361d683"
cas_config_opid_index: 6
I20260928 10:51:25.715081 12187 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } }
I20260928 10:51:25.718147 12804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Refusing update from remote peer bbaae6cf9d0245659e76c2d34361d683: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 10:51:25.718665 12891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:25.718987 12844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 10:51:25.721591 12845 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 93894c132dae473d838d5f9bf5b3fba4 (127.5.227.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:25.722726 12891 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 FOLLOWER]: Refusing update from remote peer bbaae6cf9d0245659e76c2d34361d683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 10:51:25.723821 12746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:25.725091 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.671221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.5.227.129:37621)
0928 10:51:25.687901 (+ 16680us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.687911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.687946 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:51:25.688035 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.688045 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.689263 (+ 1218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.691395 (+ 2132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.691678 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.698849 (+ 7171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.701815 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.703008 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.707183 (+ 4175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.708896 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.708911 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.708928 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.708978 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.713898 (+ 4920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.715026 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.715037 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.715041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.715247 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.715900 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.715905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.715909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.716442 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.724074 (+ 7632us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.724295 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.724634 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16433,"thread_start_us":17567,"threads_started":1}
I20260928 10:51:25.726950 12845 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.227.188:34513
I20260928 10:51:25.727417 12371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:25.727766 12758 master.cc:537] Master@127.5.227.188:34513 shutting down...
I20260928 10:51:25.727597 12804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 93894c132dae473d838d5f9bf5b3fba4 (127.5.227.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:25.732335 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:25.732896 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:25.733309 12844 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:25.733923 12844 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:25.734592 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:25.734935 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:51:25.749462 12891 raft_consensus.cc:858] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } }
W20260928 10:51:25.749676 12891 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } }
W20260928 10:51:25.768195 12870 tablet.cc:2369] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:25.777518 12891 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d827d2628e984b3195cea1e7a4786634 (127.5.227.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:25.783197 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:25.783775 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 10:51:25.784027 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbaae6cf9d0245659e76c2d34361d683. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:25.784281 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 10:51:25.783784 12948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: NON_VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: true } } }
I20260928 10:51:25.784727 12948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 10:51:25.785881 12845 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:25.786201 12845 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
W20260928 10:51:25.788020 12171 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 10:51:25.788182 12758 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 10:51:25.788753 12758 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 10:51:25.789008 12758 tablet_replica.cc:330] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: stopping tablet replica
W20260928 10:51:25.794526 12171 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 -> Peer 93894c132dae473d838d5f9bf5b3fba4 (127.5.227.188:34513): Couldn't send request to peer 93894c132dae473d838d5f9bf5b3fba4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 10:51:25.808204 12758 master.cc:552] Master@127.5.227.188:34513 shutdown complete.
I20260928 10:51:25.808323 12758 master_runner.cc:305] Clearing existing system tablet
I20260928 10:51:25.808691 12758 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:25.808822 12758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:25.811293 12758 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260928 10:51:25.813378 12954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:25.813777 12758 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:25.814105 12758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 10:51:25.814211 12758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
uuid: "93894c132dae473d838d5f9bf5b3fba4"
format_stamp: "Formatted at 2026-09-28 10:51:25 on dist-test-slave-8w8g"
server_key: "5f41eb1cfa0b096c79dc57bf94e995e1"
server_key_iv: "b33961fac96ca83b17efaf446a99f1b9"
server_key_version: "encryptionkey@0"
I20260928 10:51:25.818091 12758 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:25.823401 12758 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 10:51:25.823623 12758 log.cc:1196] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 10:51:25.824191 12758 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Deleting consensus metadata
I20260928 10:51:25.824453 12758 master_runner.cc:315] Copying system tablet from 127.5.227.190:36225
I20260928 10:51:25.827248 12758 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.227.190:36225
I20260928 10:51:25.842234 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.828171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.5.227.190:36225)
0928 10:51:25.828454 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:25.828464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:25.829136 (+ 672us) server_negotiation.cc:407] Connection header received
0928 10:51:25.829431 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:25.829437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:25.829529 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:25.829646 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:25.831032 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.831820 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.833134 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.833602 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.833750 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.835997 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:25.836007 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:25.836012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:25.836051 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:25.839211 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.840058 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.840063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.840067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.840216 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:25.840672 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:25.840678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:25.840681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:25.841047 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:25.841203 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:25.841622 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 10:51:25.841838 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 10:51:25.842268 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:25.828149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51386)
0928 10:51:25.829018 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:25.829035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:25.829225 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:25.829676 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:25.829681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:25.830137 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:25.830859 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.830871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.831984 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.831990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.832948 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:25.832960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:25.833765 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:25.833771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:25.833968 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:25.834744 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:25.834767 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:25.835701 (+ 934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:25.839375 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.839381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.839385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.839925 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.840335 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:25.840341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:25.840345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:25.840555 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:25.841162 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:25.841168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:25.841356 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 10:51:25.841506 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 10:51:25.841846 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":494,"threads_started":1}
I20260928 10:51:25.843163 12207 tablet_copy_service.cc:140] P bbaae6cf9d0245659e76c2d34361d683: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 93894c132dae473d838d5f9bf5b3fba4 ({username='oryx', principal='oryx/127.5.227.188@KRBTEST.COM'} at 127.0.0.1:51386)
I20260928 10:51:25.843335 12207 tablet_copy_service.cc:161] P bbaae6cf9d0245659e76c2d34361d683: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 93894c132dae473d838d5f9bf5b3fba4 at {username='oryx', principal='oryx/127.5.227.188@KRBTEST.COM'} at 127.0.0.1:51386: session id = 93894c132dae473d838d5f9bf5b3fba4-00000000000000000000000000000000
I20260928 10:51:25.846043 12207 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 10:51:25.847934 12758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 10:51:25.854060 12758 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: tablet copy: Starting download of 0 data blocks...
I20260928 10:51:25.854408 12758 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: tablet copy: Starting download of 1 WAL segments...
I20260928 10:51:25.856819 12758 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 10:51:25.860459 12207 tablet_copy_service.cc:342] P bbaae6cf9d0245659e76c2d34361d683: Request end of tablet copy session 93894c132dae473d838d5f9bf5b3fba4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.227.188@KRBTEST.COM'} at 127.0.0.1:51386
I20260928 10:51:25.860620 12207 tablet_copy_service.cc:433] P bbaae6cf9d0245659e76c2d34361d683: ending tablet copy session 93894c132dae473d838d5f9bf5b3fba4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 93894c132dae473d838d5f9bf5b3fba4
I20260928 10:51:25.872895 12758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:25.877801 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:25.883610 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:25.885689 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 12248
W20260928 10:51:25.892232 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:25.892817 12758 server_base.cc:1034] running on GCE node
I20260928 10:51:25.893234 12758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:25.893966 12758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:25.895181 12758 hybrid_clock.cc:648] HybridClock initialized: now 1790592685895120 us; error 56 us; skew 500 ppm
I20260928 10:51:25.895466 12758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:25 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592685, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:25.897578 12758 init.cc:377] Logged in from keytab as oryx/127.5.227.188@KRBTEST.COM (short username oryx)
I20260928 10:51:25.900738 12758 webserver.cc:460] Webserver started at http://127.5.227.188:39211/ using document root <none> and password file <none>
I20260928 10:51:25.901063 12758 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:25.901176 12758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:25.903609 12758 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 10:51:25.905831 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:25.906422 12758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 10:51:25.906574 12758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
uuid: "93894c132dae473d838d5f9bf5b3fba4"
format_stamp: "Formatted at 2026-09-28 10:51:25 on dist-test-slave-8w8g"
server_key: "5f41eb1cfa0b096c79dc57bf94e995e1"
server_key_iv: "b33961fac96ca83b17efaf446a99f1b9"
server_key_version: "encryptionkey@0"
I20260928 10:51:25.906776 12758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260928 10:51:25.910183 12172 connection.cc:535] server connection from 127.5.227.129:35389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:51:25.912292 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:37621
--local_ip_for_outbound_sockets=127.5.227.129
--tserver_master_addrs=127.5.227.190:36225
--webserver_port=36041
--webserver_interface=127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab
--principal=oryx/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:25.924553 12758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:25.928735 12758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:25.976883 12758 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.188:34513
I20260928 10:51:25.976918 13023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.188:34513 every 8 connection(s)
I20260928 10:51:25.977922 12758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
I20260928 10:51:25.986001 13024 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:51:25.992106 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Bootstrap starting.
I20260928 10:51:26.008508 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:51:26.009079 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Bootstrap complete.
I20260928 10:51:26.010113 13024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } }
I20260928 10:51:26.010360 13024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93894c132dae473d838d5f9bf5b3fba4, State: Initialized, Role: LEARNER
I20260928 10:51:26.010591 13024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } }
I20260928 10:51:26.012288 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:26.013178 13024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:26.013341 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:51:26.035514 13038 catalog_manager.cc:1259] Loaded cluster ID: ecb34afe09b145fcb8491764a29d4a70
I20260928 10:51:26.035636 13038 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: loading cluster ID for follower catalog manager: success
I20260928 10:51:26.038424 13038 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: acquiring CA information for follower catalog manager: success
I20260928 10:51:26.040406 13038 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:51:26.360412 12844 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:26.361025 12844 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:26.382242 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.360648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.0.0.1:40526)
0928 10:51:26.375086 (+ 14438us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:26.375116 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:26.375393 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:26.375989 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:26.375994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:26.376025 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:26.376184 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.376192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.378141 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.378147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.379545 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.379567 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.381588 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.381594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.381742 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.381771 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 10:51:26.381870 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 10:51:26.381991 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14250,"thread_start_us":14126,"threads_started":1}
I20260928 10:51:26.382529 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.361150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.5.227.188:34513)
0928 10:51:26.374581 (+ 13431us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:26.374593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:26.375263 (+ 670us) server_negotiation.cc:407] Connection header received
0928 10:51:26.375591 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:26.375597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:26.375709 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:26.375898 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:26.376352 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.377846 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.379866 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.381214 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.381394 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.381688 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:26.381919 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 10:51:26.382070 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13185,"thread_start_us":13054,"threads_started":1}
I20260928 10:51:26.383447 12999 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260928 10:51:26.384052 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbaae6cf9d0245659e76c2d34361d683. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: NON_VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: true } } }
I20260928 10:51:26.384385 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:51:26.384966 13040 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: attempting to promote NON_VOTER 93894c132dae473d838d5f9bf5b3fba4 to VOTER
I20260928 10:51:26.385787 13040 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } }
I20260928 10:51:26.387992 12804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Refusing update from remote peer bbaae6cf9d0245659e76c2d34361d683: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 10:51:26.388147 12999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 LEARNER]: Refusing update from remote peer bbaae6cf9d0245659e76c2d34361d683: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 10:51:26.388685 12844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 10:51:26.389252 13040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 10:51:26.393184 13040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 93894c132dae473d838d5f9bf5b3fba4 (127.5.227.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.394074 12804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 93894c132dae473d838d5f9bf5b3fba4 (127.5.227.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.396749 12999 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 93894c132dae473d838d5f9bf5b3fba4 (127.5.227.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.399724 13039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.400290 13039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:26.400759 13039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.401103 13039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:26.401468 13039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.401757 13039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:26.402737 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.403162 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:51:26.405386 13047 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbaae6cf9d0245659e76c2d34361d683" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:26.405730 13047 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 10:51:26.413647 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:26.414018 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:26.414180 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:26.424623 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:26.424823 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:26.425004 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:26.438203 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab
--principal=oryx/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:37621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=36041
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:26.440006 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:26.441501 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:26.458204 13050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:26.458703 13053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:26.458907 13051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:26.459874 12976 server_base.cc:1034] running on GCE node
I20260928 10:51:26.460669 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:26.461809 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:26.463125 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790592686463069 us; error 45 us; skew 500 ppm
I20260928 10:51:26.463598 12976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:26 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:26.468600 12976 init.cc:377] Logged in from keytab as oryx/127.5.227.129@KRBTEST.COM (short username oryx)
I20260928 10:51:26.471355 12976 webserver.cc:460] Webserver started at http://127.5.227.129:36041/ using document root <none> and password file <none>
I20260928 10:51:26.472585 12976 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:26.472810 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:26.479426 12976 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 10:51:26.483044 13060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:26.485204 12976 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260928 10:51:26.485496 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "569f926721f94d0d9d6102ae8eba1011"
format_stamp: "Formatted at 2026-09-28 10:51:22 on dist-test-slave-8w8g"
server_key: "fa62b3c284efdad0a771ae3a95e3d237"
server_key_iv: "5fea4b74b168c9bcb3dbac0cb0b0acb8"
server_key_version: "encryptionkey@0"
I20260928 10:51:26.486653 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:26.520438 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:26.525683 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:26.525986 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:26.527092 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:26.528806 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:26.528939 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:26.529064 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:26.529152 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:26.573658 12976 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:37621
I20260928 10:51:26.573725 13172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:37621 every 8 connection(s)
I20260928 10:51:26.575249 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:26.584281 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 12976
I20260928 10:51:26.584697 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 12382
Sep 28 10:51:26 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.130@KRBTEST.COM
Sep 28 10:51:26 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.189@KRBTEST.COM
Sep 28 10:51:26 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.188@KRBTEST.COM
Sep 28 10:51:26 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:26.633153 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.587453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.5.227.130:45421)
0928 10:51:26.588411 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:26.588436 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:26.604581 (+ 16145us) server_negotiation.cc:407] Connection header received
0928 10:51:26.604809 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:26.604823 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:26.605261 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:26.605504 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:26.609217 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.612062 (+ 2845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.615279 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.616352 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.616646 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.623599 (+ 6953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:26.623621 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:26.623642 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:26.623688 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:26.627521 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.628672 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.628683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.628686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.628869 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.629467 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.629473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.629477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.630052 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:26.630369 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:26.631875 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0928 10:51:26.632187 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":443,"threads_started":1}
I20260928 10:51:26.634863 12505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:26.638720 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.580141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:54755 (local address 127.5.227.189:33863)
0928 10:51:26.611491 (+ 31350us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:26.611499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:26.611519 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:51:26.611578 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:26.611583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:26.611663 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:26.611791 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:26.615515 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.616650 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.620192 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.620996 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.621146 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.629217 (+ 8071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:26.629224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:26.629229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:26.629275 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:26.634261 (+ 4986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.635721 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.635755 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.635759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.636059 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.636646 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.636652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.636655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.637015 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:26.637166 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:26.638040 (+ 874us) server_negotiation.cc:299] Negotiation successful
0928 10:51:26.638288 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31177,"thread_start_us":31006,"threads_started":1}
I20260928 10:51:26.639415 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.579322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:59201 (local address 127.5.227.190:36225)
0928 10:51:26.600662 (+ 21340us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:26.600667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:26.607779 (+ 7112us) server_negotiation.cc:407] Connection header received
0928 10:51:26.611079 (+ 3300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:26.611088 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:26.611195 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:26.611320 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:26.615094 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.616414 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.620177 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.621382 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.621489 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.629009 (+ 7520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:26.629016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:26.629020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:26.629064 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:26.633548 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.634904 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.634911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.634915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.635079 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.635939 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.635949 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.635955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.636474 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:26.636706 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:26.637860 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0928 10:51:26.638232 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21165,"thread_start_us":20933,"threads_started":1}
I20260928 10:51:26.639818 13179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.580205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.129:59201)
0928 10:51:26.607626 (+ 27421us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:26.607648 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:26.610928 (+ 3280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:26.611979 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:26.611995 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:26.613816 (+ 1821us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:26.614857 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.614879 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.616637 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.616642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.619968 (+ 3326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.619977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.622518 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.622523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.622689 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.624598 (+ 1909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:26.624623 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:26.628003 (+ 3380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:26.633763 (+ 5760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:26.633771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:26.633960 (+ 189us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:26.634706 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:26.635219 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:26.635225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:26.635229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:26.635774 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:26.636703 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:26.636716 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:26.637227 (+ 511us) client_negotiation.cc:769] Sending connection context
0928 10:51:26.637695 (+ 468us) client_negotiation.cc:240] Negotiation successful
0928 10:51:26.638365 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27217,"mutex_wait_us":615,"spinlock_wait_cycles":56960,"thread_start_us":24577,"threads_started":1}
I20260928 10:51:26.641002 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.580204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.5.227.129:54755)
0928 10:51:26.605555 (+ 25351us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:26.605649 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:26.610374 (+ 4725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:26.612013 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:26.612017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:26.613815 (+ 1798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:26.615295 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.615310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.616867 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.616872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.619923 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.619953 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.621519 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.621530 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.622582 (+ 1052us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.623674 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:26.623710 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:26.628650 (+ 4940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:26.634496 (+ 5846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:26.634501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:26.634506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:26.635218 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:26.636204 (+ 986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:26.636209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:26.636213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:26.636373 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:26.637240 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:26.637248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:26.637428 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 10:51:26.637695 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 10:51:26.640576 (+ 2881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24944,"mutex_wait_us":2786,"spinlock_wait_cycles":1280,"thread_start_us":26790,"threads_started":1}
I20260928 10:51:26.641590 13173 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36225
I20260928 10:51:26.641932 13173 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:26.642853 13173 heartbeater.cc:511] Master 127.5.227.190:36225 requested a full tablet report, sending...
I20260928 10:51:26.644361 13175 heartbeater.cc:347] Connected to a master server at 127.5.227.189:33863
I20260928 10:51:26.644541 13175 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:26.644987 13175 heartbeater.cc:511] Master 127.5.227.189:33863 requested a full tablet report, sending...
I20260928 10:51:26.645985 12187 ts_manager.cc:194] Re-registered known tserver with Master: 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621)
I20260928 10:51:26.647318 12794 ts_manager.cc:194] Registered new tserver with Master: 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621)
I20260928 10:51:26.647608 12187 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:59201
I20260928 10:51:26.649533 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.606462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:58945 (local address 127.5.227.188:34513)
0928 10:51:26.606678 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:26.606685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:26.612261 (+ 5576us) server_negotiation.cc:407] Connection header received
0928 10:51:26.612765 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:26.612772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:26.612888 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:26.612983 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:26.615188 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.616417 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.620177 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.621307 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.621426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.628162 (+ 6736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:26.628169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:26.628174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:26.628224 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:26.633134 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.641933 (+ 8799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.641944 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.641950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.642223 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:26.642771 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:26.642778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:26.642785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:26.643273 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:26.643538 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:26.648306 (+ 4768us) server_negotiation.cc:299] Negotiation successful
0928 10:51:26.649146 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 10:51:26.650642 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:26.606846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.5.227.129:58945)
0928 10:51:26.612132 (+ 5286us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:26.612160 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:26.612396 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:26.613254 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:26.613260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:26.614255 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:26.614864 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.614889 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.616748 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.616756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.619940 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:26.619960 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:26.622432 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:26.622445 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:26.623151 (+ 706us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:26.623838 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:26.623910 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:26.627800 (+ 3890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:26.640925 (+ 13125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:26.640929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:26.640932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:26.641752 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:26.642388 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:26.642393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:26.642397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:26.642621 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:26.643427 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:26.643432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:26.643607 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 10:51:26.643900 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 10:51:26.644218 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4999,"mutex_wait_us":1016,"thread_start_us":5537,"threads_started":1}
I20260928 10:51:26.651808 13174 heartbeater.cc:347] Connected to a master server at 127.5.227.188:34513
I20260928 10:51:26.651963 13174 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:26.652354 13174 heartbeater.cc:511] Master 127.5.227.188:34513 requested a full tablet report, sending...
I20260928 10:51:26.654680 12989 ts_manager.cc:194] Registered new tserver with Master: 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621)
I20260928 10:51:26.739246 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 12382
I20260928 10:51:26.757117 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:45421
--local_ip_for_outbound_sockets=127.5.227.130
--tserver_master_addrs=127.5.227.190:36225
--webserver_port=42695
--webserver_interface=127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab
--principal=oryx/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:27.144977 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:27.145303 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:27.145478 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:27.154887 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:27.155239 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:27.155943 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:27.167860 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab
--principal=oryx/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:45421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=42695
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:27.169358 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:27.170897 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:27.188668 13190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:27.188747 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:27.188714 13193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:27.189291 13185 server_base.cc:1034] running on GCE node
I20260928 10:51:27.190042 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:27.191206 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:27.192433 13185 hybrid_clock.cc:648] HybridClock initialized: now 1790592687192308 us; error 112 us; skew 500 ppm
I20260928 10:51:27.192879 13185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:27 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:27.203799 13185 init.cc:377] Logged in from keytab as oryx/127.5.227.130@KRBTEST.COM (short username oryx)
I20260928 10:51:27.205868 13185 webserver.cc:460] Webserver started at http://127.5.227.130:42695/ using document root <none> and password file <none>
I20260928 10:51:27.207005 13185 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:27.207199 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:27.213685 13185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 10:51:27.217336 13200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:27.219275 13185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:51:27.219601 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "5b7f2fa7b49884d32477ac0492e03b5a"
server_key_iv: "9b52c2de2ceb014929357f0cf1e01dd2"
server_key_version: "encryptionkey@0"
I20260928 10:51:27.220758 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:27.237116 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:27.243000 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:27.243333 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:27.244653 13185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:27.246238 13185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:27.246385 13185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:27.246519 13185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:27.246611 13185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:27.285096 13185 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:45421
I20260928 10:51:27.285193 13312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:45421 every 8 connection(s)
I20260928 10:51:27.286661 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:27.293568 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 13185
I20260928 10:51:27.293933 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 12516
Sep 28 10:51:27 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.189@KRBTEST.COM
Sep 28 10:51:27 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.188@KRBTEST.COM
Sep 28 10:51:27 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
Sep 28 10:51:27 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.131@KRBTEST.COM
I20260928 10:51:27.323875 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.293717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:47951 (local address 127.5.227.188:34513)
0928 10:51:27.295116 (+ 1399us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:27.295123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:27.295145 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:51:27.297038 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:27.297044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:27.297129 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:27.297229 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:27.300408 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.301165 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.304194 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.304721 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.304803 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.312030 (+ 7227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:27.312052 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:27.312059 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:27.312111 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:27.318055 (+ 5944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.319346 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.319352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.319356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.319546 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.320133 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.320138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.320142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.320534 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:27.320677 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:27.323134 (+ 2457us) server_negotiation.cc:299] Negotiation successful
0928 10:51:27.323430 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258,"thread_start_us":983,"threads_started":1}
I20260928 10:51:27.325521 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.293320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.5.227.130:47951)
0928 10:51:27.294489 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:27.294508 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:27.296565 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:27.297414 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:27.297418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:27.298887 (+ 1469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:27.300154 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.300178 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.301348 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.301354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.303997 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.304013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.305174 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.305180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.305681 (+ 501us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.306759 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:27.306805 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:27.311693 (+ 4888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:27.318299 (+ 6606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:27.318315 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:27.318342 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:27.319164 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:27.319686 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:27.319692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:27.319696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:27.319993 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:27.320754 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:27.320763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:27.321232 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 10:51:27.321569 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 10:51:27.322138 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":265,"spinlock_wait_cycles":21760,"thread_start_us":834,"threads_started":1}
I20260928 10:51:27.326141 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.295206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.5.227.131:33795)
0928 10:51:27.296073 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:27.296083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:27.296113 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:51:27.296186 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:27.296196 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:27.296496 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:27.296675 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:27.299700 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.301520 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.303496 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.305437 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.305591 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.316180 (+ 10589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:27.316207 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:27.316231 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:27.316301 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:27.321096 (+ 4795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.322073 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.322083 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.322086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.322249 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.322777 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.322781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.322783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.323191 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:27.323541 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:27.324168 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 10:51:27.324985 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":524,"threads_started":1}
I20260928 10:51:27.327061 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.290751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.130:42111)
0928 10:51:27.293038 (+ 2287us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:27.293061 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:27.296565 (+ 3504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:27.297355 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:27.297369 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:27.298887 (+ 1518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:27.300154 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.300178 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.301617 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.301623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.303997 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.304013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.305227 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.305234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.306168 (+ 934us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.307417 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:27.307445 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:27.312396 (+ 4951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:27.322713 (+ 10317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:27.322720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:27.322724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:27.323492 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:27.324568 (+ 1076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:27.324572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:27.324576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:27.324738 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:27.325546 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:27.325551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:27.325910 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 10:51:27.326102 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 10:51:27.326591 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2100,"mutex_wait_us":1841,"thread_start_us":935,"threads_started":1}
I20260928 10:51:27.327095 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.290099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:42111 (local address 127.5.227.190:36225)
0928 10:51:27.293526 (+ 3427us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:27.293532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:27.293555 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:51:27.296815 (+ 3260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:27.296821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:27.296916 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:27.297040 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:27.300371 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.301454 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.304154 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.304807 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.304943 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.313242 (+ 8299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:27.313249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:27.313253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:27.313300 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:27.317292 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.323651 (+ 6359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.323660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.323666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.323878 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.324943 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.324951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.324954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.325408 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:27.325578 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:27.326385 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 10:51:27.326689 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3293,"thread_start_us":3724,"threads_started":1}
I20260928 10:51:27.327633 12639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:27.327878 13314 heartbeater.cc:347] Connected to a master server at 127.5.227.188:34513
I20260928 10:51:27.328343 13314 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:27.329473 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.290094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:35267 (local address 127.5.227.189:33863)
0928 10:51:27.291343 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:27.291350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:27.292259 (+ 909us) server_negotiation.cc:407] Connection header received
0928 10:51:27.296853 (+ 4594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:27.296860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:27.296942 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:27.297028 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:27.301703 (+ 4675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.302881 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.304944 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.305633 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.305762 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.311098 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:27.311104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:27.311107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:27.311143 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:27.318318 (+ 7175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.324584 (+ 6266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.324594 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.324597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.324784 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:27.325211 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:27.325218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:27.325222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:27.325617 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:27.325793 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:27.328869 (+ 3076us) server_negotiation.cc:299] Negotiation successful
0928 10:51:27.329112 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067,"thread_start_us":902,"threads_started":1}
I20260928 10:51:27.329555 13314 heartbeater.cc:511] Master 127.5.227.188:34513 requested a full tablet report, sending...
I20260928 10:51:27.329663 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.290751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.5.227.130:35267)
0928 10:51:27.292064 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:27.292115 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:27.296475 (+ 4360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:27.297356 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:27.297369 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:27.298904 (+ 1535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:27.300146 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.300180 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.303102 (+ 2922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.303108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.304775 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.304790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.305890 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.305897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.306069 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.306912 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:27.306962 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:27.310782 (+ 3820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:27.323721 (+ 12939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:27.323745 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:27.323749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:27.324429 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:27.324913 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:27.324917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:27.324921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:27.325098 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:27.325748 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:27.325753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:27.327691 (+ 1938us) client_negotiation.cc:769] Sending connection context
0928 10:51:27.328684 (+ 993us) client_negotiation.cc:240] Negotiation successful
0928 10:51:27.329035 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":159,"spinlock_wait_cycles":50816,"thread_start_us":677,"threads_started":1}
I20260928 10:51:27.331156 13313 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36225
I20260928 10:51:27.331338 13313 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:27.331929 13313 heartbeater.cc:511] Master 127.5.227.190:36225 requested a full tablet report, sending...
I20260928 10:51:27.332132 12989 ts_manager.cc:194] Registered new tserver with Master: fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421)
I20260928 10:51:27.332448 13315 heartbeater.cc:347] Connected to a master server at 127.5.227.189:33863
I20260928 10:51:27.332609 13315 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:27.333132 13315 heartbeater.cc:511] Master 127.5.227.189:33863 requested a full tablet report, sending...
I20260928 10:51:27.333241 12187 ts_manager.cc:194] Re-registered known tserver with Master: fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421)
I20260928 10:51:27.334605 12187 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:42111
I20260928 10:51:27.335642 12794 ts_manager.cc:194] Registered new tserver with Master: fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421)
I20260928 10:51:27.424227 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 12516
I20260928 10:51:27.443779 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:33795
--local_ip_for_outbound_sockets=127.5.227.131
--tserver_master_addrs=127.5.227.190:36225
--webserver_port=46545
--webserver_interface=127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab
--principal=oryx/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:27.653083 13173 heartbeater.cc:503] Master 127.5.227.190:36225 was elected leader, sending a full tablet report...
W20260928 10:51:27.810678 13326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:27.811033 13326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:27.811183 13326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:27.819700 13326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:27.819965 13326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:27.820143 13326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:27.831898 13326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab
--principal=oryx/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:33795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=46545
--tserver_master_addrs=127.5.227.190:36225,127.5.227.189:33863,127.5.227.188:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:27.833348 13326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:27.834775 13326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:27.849789 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:27.849725 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:27.849757 13334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:27.851361 13326 server_base.cc:1034] running on GCE node
I20260928 10:51:27.852144 13326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:27.853256 13326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:27.854677 13326 hybrid_clock.cc:648] HybridClock initialized: now 1790592687854631 us; error 94 us; skew 500 ppm
I20260928 10:51:27.855118 13326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:27 dist-test-slave-8w8g krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:27.859503 13326 init.cc:377] Logged in from keytab as oryx/127.5.227.131@KRBTEST.COM (short username oryx)
I20260928 10:51:27.861251 13326 webserver.cc:460] Webserver started at http://127.5.227.131:46545/ using document root <none> and password file <none>
I20260928 10:51:27.861986 13326 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:27.862114 13326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:27.868978 13326 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 10:51:27.872466 13341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:27.874297 13326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:51:27.874570 13326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "6458de1cef7a42e6b3344e806939474d"
format_stamp: "Formatted at 2026-09-28 10:51:23 on dist-test-slave-8w8g"
server_key: "02c9f4b6e5992890a4d54c05d07f31e4"
server_key_iv: "9468db3624ff1618c89a771d2d0175d4"
server_key_version: "encryptionkey@0"
I20260928 10:51:27.875877 13326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:27.898276 13326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:27.904256 13326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:27.904599 13326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:27.905774 13326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:27.907538 13326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:27.907630 13326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:27.907807 13326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:27.907893 13326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:27.949122 13326 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:33795
I20260928 10:51:27.949168 13453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:33795 every 8 connection(s)
I20260928 10:51:27.951076 13326 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:27.959558 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 13326
Sep 28 10:51:27 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.189@KRBTEST.COM
Sep 28 10:51:28 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.188@KRBTEST.COM
I20260928 10:51:28.018301 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.954359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:57531 (local address 127.5.227.188:34513)
0928 10:51:27.967203 (+ 12844us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:27.967213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:27.967244 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:51:27.972810 (+ 5566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:27.972821 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:27.972965 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:27.973189 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:27.993687 (+ 20498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.995803 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.998818 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.999646 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.999842 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.006579 (+ 6737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.006590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.006598 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.006659 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.011786 (+ 5127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.013316 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.013326 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.013332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.013534 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.014308 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.014316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.014322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.014827 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.015021 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.017171 (+ 2150us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.017740 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12677,"thread_start_us":12509,"threads_started":1}
I20260928 10:51:28.020108 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.954371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:50971 (local address 127.5.227.189:33863)
0928 10:51:27.968135 (+ 13764us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:27.968142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:27.968165 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:51:27.973646 (+ 5481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:27.973658 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:27.973783 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:27.973895 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:27.978459 (+ 4564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.980179 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.984269 (+ 4090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.986786 (+ 2517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.986943 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.004294 (+ 17351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.004307 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.004316 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.004402 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.013281 (+ 8879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.015184 (+ 1903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.015194 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.015200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.015445 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.017012 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.017024 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.017031 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.017775 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.018061 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.018936 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.019311 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13648,"thread_start_us":13552,"threads_started":1}
I20260928 10:51:28.021848 13460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.955260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.5.227.131:50971)
0928 10:51:27.967913 (+ 12653us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:27.967934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:27.973412 (+ 5478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:27.974638 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:27.974664 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:27.976930 (+ 2266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:27.978161 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.978198 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.980521 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.980529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.983828 (+ 3299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.983863 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.993051 (+ 9188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.993065 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.993608 (+ 543us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:27.995901 (+ 2293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:27.995955 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:28.002303 (+ 6348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:28.013541 (+ 11238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:28.013550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:28.013558 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:28.014508 (+ 950us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:28.015678 (+ 1170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:28.015688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:28.015696 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:28.016728 (+ 1032us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:28.018232 (+ 1504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:28.018241 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:28.018485 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.020622 (+ 2137us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.021186 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12504,"spinlock_wait_cycles":84352,"thread_start_us":9860,"threads_started":1}
I20260928 10:51:28.021375 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.955081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.5.227.131:57531)
0928 10:51:27.965845 (+ 10764us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:27.965933 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:27.972014 (+ 6081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:27.975898 (+ 3884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:27.975908 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:27.992676 (+ 16768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:27.993216 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.993237 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:27.996186 (+ 2949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:27.996195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:27.998553 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:27.998578 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.000348 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.000358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.000629 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.002748 (+ 2119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:28.002783 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:28.006218 (+ 3435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:28.012043 (+ 5825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:28.012059 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:28.012088 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:28.013100 (+ 1012us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:28.013703 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:28.013711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:28.013717 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:28.014104 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:28.015141 (+ 1037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:28.015156 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:28.016530 (+ 1374us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.018795 (+ 2265us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.019575 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10214,"spinlock_wait_cycles":1252864,"thread_start_us":12302,"threads_started":1}
I20260928 10:51:28.025008 13456 heartbeater.cc:347] Connected to a master server at 127.5.227.189:33863
I20260928 10:51:28.025342 13455 heartbeater.cc:347] Connected to a master server at 127.5.227.188:34513
I20260928 10:51:28.025458 13456 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:28.025487 13455 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:28.026654 13455 heartbeater.cc:511] Master 127.5.227.188:34513 requested a full tablet report, sending...
I20260928 10:51:28.027496 13456 heartbeater.cc:511] Master 127.5.227.189:33863 requested a full tablet report, sending...
I20260928 10:51:28.029035 12989 ts_manager.cc:194] Registered new tserver with Master: 6458de1cef7a42e6b3344e806939474d (127.5.227.131:33795)
I20260928 10:51:28.029026 12794 ts_manager.cc:194] Registered new tserver with Master: 6458de1cef7a42e6b3344e806939474d (127.5.227.131:33795)
Sep 28 10:51:28 dist-test-slave-8w8g krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592687, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:28.046855 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.011247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.5.227.188:34513)
0928 10:51:28.018685 (+ 7438us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.018693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.023602 (+ 4909us) server_negotiation.cc:407] Connection header received
0928 10:51:28.024277 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.024285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.024378 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.024571 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.028381 (+ 3810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.029706 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.032144 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.033075 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.033193 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.039197 (+ 6004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.039205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.039210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.039254 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.042755 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.043517 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.043562 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.043566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.043774 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.045050 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.045055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.045062 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.045456 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.045726 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.046152 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.046416 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7326,"thread_start_us":18184,"threads_started":1}
I20260928 10:51:28.049957 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.968201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.131:53231)
0928 10:51:27.975107 (+ 6906us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:27.975138 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:27.975325 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:28.010109 (+ 34784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:28.010119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:28.010913 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:28.011310 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.011332 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.022804 (+ 11472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.022810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.025445 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.025470 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.026775 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.026782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.027106 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.030043 (+ 2937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:28.030075 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:28.034717 (+ 4642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:28.041948 (+ 7231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:28.041954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:28.041958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:28.042530 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:28.044254 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:28.044264 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:28.044268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:28.044499 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:28.048279 (+ 3780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:28.048285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:28.048781 (+ 496us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.048964 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.049323 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6668,"thread_start_us":7238,"threads_started":1}
I20260928 10:51:28.050235 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.984911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.5.227.189:33863)
0928 10:51:28.015021 (+ 30110us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.015031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.015058 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:51:28.024512 (+ 9454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.024517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.024595 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.024670 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.031998 (+ 7328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.033217 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.037430 (+ 4213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.038136 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.038248 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.042354 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.042362 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.042367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.042569 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.046740 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.047642 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.047649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.047653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.047875 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.048349 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.048353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.048356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.048745 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.048993 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.049592 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.049844 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29874,"thread_start_us":20311,"threads_started":1}
I20260928 10:51:28.051102 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.968318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:53231 (local address 127.5.227.190:36225)
0928 10:51:28.009616 (+ 41298us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.009621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.009642 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:51:28.009713 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.009717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.009840 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.009983 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.019601 (+ 9618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.022437 (+ 2836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.025808 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.026548 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.026712 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.035583 (+ 8871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.035593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.035600 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.035648 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.041733 (+ 6085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.043885 (+ 2152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.043891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.043894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.044079 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.045197 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.045202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.045205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.048081 (+ 2876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.050294 (+ 2213us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.050479 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.050668 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41141,"thread_start_us":64937,"threads_started":1}
I20260928 10:51:28.052418 13454 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36225
I20260928 10:51:28.052603 13454 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:28.053064 13454 heartbeater.cc:511] Master 127.5.227.190:36225 requested a full tablet report, sending...
I20260928 10:51:28.054646 12187 ts_manager.cc:194] Re-registered known tserver with Master: 6458de1cef7a42e6b3344e806939474d (127.5.227.131:33795)
I20260928 10:51:28.056255 12187 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:53231
I20260928 10:51:28.057487 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:27.978213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.5.227.190:36225)
0928 10:51:28.032672 (+ 54459us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.032679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.032707 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:51:28.032775 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.032781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.032879 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.033102 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.036505 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.038054 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.042259 (+ 4205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.044491 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.044688 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.048468 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.048475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.048480 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.048520 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.053793 (+ 5273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.054604 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.054610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.054613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.054823 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.055490 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.055495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.055498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.055929 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.056147 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.056894 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.057176 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54106,"thread_start_us":31906,"threads_started":1}
I20260928 10:51:28.062474 12187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:51:28.070226 12187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:28.126348 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.104527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.5.227.129:37621)
0928 10:51:28.112513 (+ 7986us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.112521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.113209 (+ 688us) server_negotiation.cc:407] Connection header received
0928 10:51:28.113565 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.113576 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.113999 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.114960 (+ 961us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:28.115637 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.117822 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.122638 (+ 4816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.124841 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.125023 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.125704 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.125784 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.126042 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7750,"thread_start_us":7587,"threads_started":1}
I20260928 10:51:28.126403 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.104256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37621 (local address 127.0.0.1:43056)
0928 10:51:28.112957 (+ 8701us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:28.112980 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:28.113249 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:28.114910 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:28.114914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:28.114942 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:28.115333 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.115346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.118173 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.118183 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.122321 (+ 4138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.122350 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.125089 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.125097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.125269 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.125310 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.125408 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.125708 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8527,"thread_start_us":8372,"threads_started":1}
I20260928 10:51:28.127863 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.113094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:33795 (local address 127.0.0.1:58902)
0928 10:51:28.115845 (+ 2751us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:28.115873 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:28.115997 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:28.117577 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:28.117585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:28.117629 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:28.118182 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.118200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.121557 (+ 3357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.121567 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.123926 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.123938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.127044 (+ 3106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.127050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.127242 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.127280 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.127347 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.127528 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2561,"thread_start_us":2382,"threads_started":1}
I20260928 10:51:28.131124 13108 tablet_service.cc:1467] Processing CreateTablet for tablet 02028269f7f34af58f26dca7bcf3470d (DEFAULT_TABLE table=test-table [id=ec64838fe32646608485612fc4214476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:28.132530 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.113579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.5.227.131:33795)
0928 10:51:28.114348 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.114360 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.116292 (+ 1932us) server_negotiation.cc:407] Connection header received
0928 10:51:28.116476 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.116491 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.116867 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.117148 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:28.118534 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.121245 (+ 2711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.124081 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.126813 (+ 2732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.128183 (+ 1370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.131889 (+ 3706us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.131980 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.132190 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":894,"threads_started":1}
I20260928 10:51:28.132742 13108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02028269f7f34af58f26dca7bcf3470d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:28.133252 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.116064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:45421 (local address 127.0.0.1:53604)
0928 10:51:28.122790 (+ 6726us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:28.122813 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:28.123261 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:28.125403 (+ 2142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:28.125408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:28.125435 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:28.126349 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.126362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.129273 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.129281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.130789 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.130805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.132304 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.132312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.132532 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.132578 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.132697 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.132871 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6533,"thread_start_us":6921,"threads_started":1}
I20260928 10:51:28.134143 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.116166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.5.227.130:45421)
0928 10:51:28.124533 (+ 8367us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.124545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.124577 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:51:28.124694 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.124704 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.125039 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.125258 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:28.127480 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.129137 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.130971 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.132032 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.132179 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.132373 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.133598 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.133812 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8187,"thread_start_us":8051,"threads_started":1}
I20260928 10:51:28.136649 13389 tablet_service.cc:1467] Processing CreateTablet for tablet 02028269f7f34af58f26dca7bcf3470d (DEFAULT_TABLE table=test-table [id=ec64838fe32646608485612fc4214476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:28.138271 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02028269f7f34af58f26dca7bcf3470d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:28.138626 13248 tablet_service.cc:1467] Processing CreateTablet for tablet 02028269f7f34af58f26dca7bcf3470d (DEFAULT_TABLE table=test-table [id=ec64838fe32646608485612fc4214476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:28.140129 13248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02028269f7f34af58f26dca7bcf3470d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:28.147965 13491 tablet_bootstrap.cc:492] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Bootstrap starting.
I20260928 10:51:28.151690 13491 tablet_bootstrap.cc:654] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:28.153285 13491 log.cc:824] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:28.156873 13491 tablet_bootstrap.cc:492] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: No bootstrap required, opened a new log
I20260928 10:51:28.157224 13491 ts_tablet_manager.cc:1397] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 10:51:28.160461 13493 tablet_bootstrap.cc:492] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Bootstrap starting.
I20260928 10:51:28.162801 13494 tablet_bootstrap.cc:492] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Bootstrap starting.
I20260928 10:51:28.165813 13493 tablet_bootstrap.cc:654] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:28.166601 13494 tablet_bootstrap.cc:654] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:28.167446 13493 log.cc:824] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:28.168038 13494 log.cc:824] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:28.168535 13491 raft_consensus.cc:348] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.169226 13491 raft_consensus.cc:374] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:28.169335 13491 raft_consensus.cc:729] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569f926721f94d0d9d6102ae8eba1011, State: Initialized, Role: FOLLOWER
I20260928 10:51:28.170264 13491 consensus_queue.cc:260] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.173817 13491 ts_tablet_manager.cc:1428] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260928 10:51:28.177263 13493 tablet_bootstrap.cc:492] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: No bootstrap required, opened a new log
I20260928 10:51:28.177376 13494 tablet_bootstrap.cc:492] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: No bootstrap required, opened a new log
I20260928 10:51:28.177824 13494 ts_tablet_manager.cc:1397] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.009s
I20260928 10:51:28.178194 13493 ts_tablet_manager.cc:1397] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20260928 10:51:28.190860 13494 raft_consensus.cc:348] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.191437 13494 raft_consensus.cc:374] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:28.191579 13494 raft_consensus.cc:729] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdf8c9f9d7d484a9f8fb96bfb632148, State: Initialized, Role: FOLLOWER
I20260928 10:51:28.192376 13494 consensus_queue.cc:260] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.193607 13313 heartbeater.cc:503] Master 127.5.227.190:36225 was elected leader, sending a full tablet report...
I20260928 10:51:28.194447 13494 ts_tablet_manager.cc:1428] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260928 10:51:28.195812 13493 raft_consensus.cc:348] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.196910 13493 raft_consensus.cc:374] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:28.197039 13493 raft_consensus.cc:729] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6458de1cef7a42e6b3344e806939474d, State: Initialized, Role: FOLLOWER
I20260928 10:51:28.198074 13493 consensus_queue.cc:260] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.200171 13454 heartbeater.cc:503] Master 127.5.227.190:36225 was elected leader, sending a full tablet report...
I20260928 10:51:28.200505 13493 ts_tablet_manager.cc:1428] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Time spent starting tablet: real 0.022s user 0.017s sys 0.001s
W20260928 10:51:28.204630 13457 tablet.cc:2369] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:51:28.290742 13316 tablet.cc:2369] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:28.306666 13499 raft_consensus.cc:482] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:28.306972 13499 raft_consensus.cc:504] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.309326 13499 leader_election.cc:288] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621), fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421)
I20260928 10:51:28.318854 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.310392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:56979 (local address 127.5.227.130:45421)
0928 10:51:28.310752 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.310758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.310777 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:51:28.310846 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.310850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.310917 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.311024 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:28.311651 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.312900 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.315141 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.317341 (+ 2200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.317573 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.317656 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.318336 (+ 680us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.318590 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260928 10:51:28.318804 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.309777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37621 (local address 127.5.227.131:53887)
0928 10:51:28.310080 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:28.310104 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:28.310291 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:28.310900 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:28.310904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:28.310933 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:28.311329 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.311342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.313177 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.313181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.314919 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.314936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.317655 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.317664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.318078 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.318129 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.318247 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.318446 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":182}
I20260928 10:51:28.318854 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.309871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:53887 (local address 127.5.227.129:37621)
0928 10:51:28.310106 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.310113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.310187 (+ 74us) server_negotiation.cc:407] Connection header received
0928 10:51:28.310660 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.310666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.310750 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.310853 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:28.311651 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.312947 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.315180 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.317339 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.317587 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.317655 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.318336 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.318590 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:51:28.318804 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.310283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:45421 (local address 127.5.227.131:56979)
0928 10:51:28.310469 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:28.310485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:28.310563 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:28.311032 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:28.311036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:28.311060 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:28.311330 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.311342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.313170 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.313177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.314917 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:28.314936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.317655 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.317663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:28.317962 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.318123 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 10:51:28.318247 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 10:51:28.318446 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":262}
I20260928 10:51:28.320566 13268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02028269f7f34af58f26dca7bcf3470d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" is_pre_election: true
I20260928 10:51:28.320549 13128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02028269f7f34af58f26dca7bcf3470d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569f926721f94d0d9d6102ae8eba1011" is_pre_election: true
I20260928 10:51:28.321192 13128 raft_consensus.cc:2454] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 0.
I20260928 10:51:28.321192 13268 raft_consensus.cc:2454] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 0.
I20260928 10:51:28.322269 13343 leader_election.cc:302] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569f926721f94d0d9d6102ae8eba1011, 6458de1cef7a42e6b3344e806939474d; no voters:
I20260928 10:51:28.322762 13499 raft_consensus.cc:2781] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:28.322954 13499 raft_consensus.cc:482] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:28.323072 13499 raft_consensus.cc:3037] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:28.326356 13499 raft_consensus.cc:504] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.327139 13499 leader_election.cc:288] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 election: Requested vote from peers 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621), fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421)
I20260928 10:51:28.327864 13128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02028269f7f34af58f26dca7bcf3470d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569f926721f94d0d9d6102ae8eba1011"
I20260928 10:51:28.328156 13128 raft_consensus.cc:3037] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:28.328398 13268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02028269f7f34af58f26dca7bcf3470d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148"
I20260928 10:51:28.328925 13268 raft_consensus.cc:3037] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Advancing to term 1
W20260928 10:51:28.331081 13176 tablet.cc:2369] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:28.331845 13268 raft_consensus.cc:2454] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 1.
I20260928 10:51:28.331835 13128 raft_consensus.cc:2454] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 1.
I20260928 10:51:28.332440 13343 leader_election.cc:302] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6458de1cef7a42e6b3344e806939474d, fcdf8c9f9d7d484a9f8fb96bfb632148; no voters:
I20260928 10:51:28.332816 13499 raft_consensus.cc:2781] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:28.333357 13499 raft_consensus.cc:686] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 1 LEADER]: Becoming Leader. State: Replica: 6458de1cef7a42e6b3344e806939474d, State: Running, Role: LEADER
I20260928 10:51:28.334072 13499 consensus_queue.cc:237] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } }
I20260928 10:51:28.340421 12187 catalog_manager.cc:5495] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6458de1cef7a42e6b3344e806939474d (127.5.227.131). New cstate: current_term: 1 leader_uuid: "6458de1cef7a42e6b3344e806939474d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } health_report { overall_health: UNKNOWN } } }
I20260928 10:51:28.419555 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.412901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.5.227.131:33795)
0928 10:51:28.413165 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.413172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.413321 (+ 149us) server_negotiation.cc:407] Connection header received
0928 10:51:28.413617 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.413624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.413722 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.413908 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:28.414328 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.415098 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.416754 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.417477 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.417688 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.418292 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:28.418864 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:28.419053 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.419157 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.419256 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 10:51:28.429576 13128 raft_consensus.cc:1261] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Refusing update from remote peer 6458de1cef7a42e6b3344e806939474d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:28.429637 13268 raft_consensus.cc:1261] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Refusing update from remote peer 6458de1cef7a42e6b3344e806939474d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:28.431275 13500 consensus_queue.cc:1035] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:28.431792 13499 consensus_queue.cc:1035] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [LEADER]: Connected to new peer: Peer: permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:28.450587 13503 mvcc.cc:205] Tried to move back new op lower bound from 7334267651796004864 to 7334267651422150656. Current Snapshot: MvccSnapshot[applied={T|T < 7334267651796004864}]
I20260928 10:51:28.453651 13505 mvcc.cc:205] Tried to move back new op lower bound from 7334267651796004864 to 7334267651422150656. Current Snapshot: MvccSnapshot[applied={T|T < 7334267651796004864}]
I20260928 10:51:28.466203 13504 mvcc.cc:205] Tried to move back new op lower bound from 7334267651796004864 to 7334267651422150656. Current Snapshot: MvccSnapshot[applied={T|T < 7334267651796004864}]
I20260928 10:51:28.479701 12187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:28.480142 12187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:28.487694 12187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbaae6cf9d0245659e76c2d34361d683: Sending DeleteTablet for 3 replicas of tablet 02028269f7f34af58f26dca7bcf3470d
I20260928 10:51:28.489185 13389 tablet_service.cc:1514] Processing DeleteTablet for tablet 02028269f7f34af58f26dca7bcf3470d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:28 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:58902
I20260928 10:51:28.489300 13248 tablet_service.cc:1514] Processing DeleteTablet for tablet 02028269f7f34af58f26dca7bcf3470d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:28 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:53604
I20260928 10:51:28.490712 13108 tablet_service.cc:1514] Processing DeleteTablet for tablet 02028269f7f34af58f26dca7bcf3470d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:28 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:43056
I20260928 10:51:28.495711 13513 tablet_replica.cc:330] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: stopping tablet replica
I20260928 10:51:28.496474 13513 raft_consensus.cc:2229] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:28.497434 13513 raft_consensus.cc:2258] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:28.498521 13512 tablet_replica.cc:330] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: stopping tablet replica
I20260928 10:51:28.499025 13512 raft_consensus.cc:2229] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:28.499830 13512 raft_consensus.cc:2258] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:28.500617 13513 ts_tablet_manager.cc:1905] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:28.500849 13514 tablet_replica.cc:330] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: stopping tablet replica
I20260928 10:51:28.501354 13514 raft_consensus.cc:2229] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:28.501904 13514 raft_consensus.cc:2258] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:28.502197 13512 ts_tablet_manager.cc:1905] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:28.504665 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 12155
I20260928 10:51:28.504966 13514 ts_tablet_manager.cc:1905] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:28.505883 12217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:28.511976 13512 ts_tablet_manager.cc:1918] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:28.512224 13512 log.cc:1196] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/02028269f7f34af58f26dca7bcf3470d
I20260928 10:51:28.513108 13512 ts_tablet_manager.cc:1939] T 02028269f7f34af58f26dca7bcf3470d P 6458de1cef7a42e6b3344e806939474d: Deleting consensus metadata
I20260928 10:51:28.515760 12172 catalog_manager.cc:4841] TS 6458de1cef7a42e6b3344e806939474d (127.5.227.131:33795): tablet 02028269f7f34af58f26dca7bcf3470d (table test-table [id=ec64838fe32646608485612fc4214476]) successfully deleted
I20260928 10:51:28.516855 13514 ts_tablet_manager.cc:1918] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:28.517189 13514 log.cc:1196] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/02028269f7f34af58f26dca7bcf3470d
I20260928 10:51:28.518117 13514 ts_tablet_manager.cc:1939] T 02028269f7f34af58f26dca7bcf3470d P 569f926721f94d0d9d6102ae8eba1011: Deleting consensus metadata
I20260928 10:51:28.521504 13513 ts_tablet_manager.cc:1918] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:28.521770 13513 log.cc:1196] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/02028269f7f34af58f26dca7bcf3470d
I20260928 10:51:28.523133 13513 ts_tablet_manager.cc:1939] T 02028269f7f34af58f26dca7bcf3470d P fcdf8c9f9d7d484a9f8fb96bfb632148: Deleting consensus metadata
I20260928 10:51:28.540598 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.491851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.5.227.189:33863)
0928 10:51:28.492063 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.492072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.502883 (+ 10811us) server_negotiation.cc:407] Connection header received
0928 10:51:28.503144 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.503151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.503241 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.503347 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.517552 (+ 14205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.518687 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.526286 (+ 7599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.526971 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.527098 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.531967 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.531975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.531981 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.532035 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.536305 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.537670 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.537679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.537683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.537886 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.538452 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.538460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.538466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.538971 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.539274 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.539808 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.540111 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 10:51:28.561311 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.503680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.5.227.188:34513)
0928 10:51:28.503895 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.503902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.540603 (+ 36701us) server_negotiation.cc:407] Connection header received
0928 10:51:28.540912 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.540919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.541008 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.541083 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.543387 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.544810 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.547417 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.548223 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.548391 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.552204 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.552213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.552219 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.552273 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.556429 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.557680 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.557687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.557692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.557866 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.558393 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.558399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.558404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.559045 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.559429 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.560657 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.560930 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:51:28.668387 12172 catalog_manager.cc:4841] TS 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621): tablet 02028269f7f34af58f26dca7bcf3470d (table test-table [id=ec64838fe32646608485612fc4214476]) successfully deleted
W20260928 10:51:28.668663 12172 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.5.227.129:37621, user_credentials={real_user=slave}} blocked reactor thread for 147812us
I20260928 10:51:28.669229 12172 catalog_manager.cc:4841] TS fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421): tablet 02028269f7f34af58f26dca7bcf3470d (table test-table [id=ec64838fe32646608485612fc4214476]) successfully deleted
I20260928 10:51:28.670205 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 12155
I20260928 10:51:28.725262 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.701510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.5.227.189:33863)
0928 10:51:28.701739 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.701747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.704764 (+ 3017us) server_negotiation.cc:407] Connection header received
0928 10:51:28.705018 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.705025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.705109 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.705208 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.708453 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.709871 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.711787 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.712316 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.712636 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.717202 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.717211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.717218 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.717266 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.721718 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.722629 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.722636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.722640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.722794 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.723227 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.723233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.723237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.723635 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.723917 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.724590 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.724878 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:51:28.725410 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:28.703700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.5.227.188:34513)
0928 10:51:28.703991 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:28.703998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:28.704683 (+ 685us) server_negotiation.cc:407] Connection header received
0928 10:51:28.704957 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:28.704965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:28.705056 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:28.705208 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:28.708698 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.709997 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.712371 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:28.713078 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:28.713257 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:28.717085 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:28.717095 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:28.717100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:28.717154 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:28.721141 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.722104 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.722111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.722114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.722288 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:28.722755 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:28.722761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:28.722765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:28.723233 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:28.723517 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:28.724573 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0928 10:51:28.724982 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 10:51:29.466913 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.465987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.131:51763)
0928 10:51:29.466523 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.466682 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":273,"threads_started":1}
W20260928 10:51:29.467476 13454 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:36225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:29.469748 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.468943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.130:42899)
0928 10:51:29.469469 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.469585 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":289,"threads_started":1}
W20260928 10:51:29.470167 13313 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:36225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:29.472544 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.471674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.129:56493)
0928 10:51:29.472231 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.472378 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":362,"threads_started":1}
W20260928 10:51:29.473101 13173 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:36225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:29.988456 13526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbaae6cf9d0245659e76c2d34361d683)
I20260928 10:51:29.988768 13526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } }
I20260928 10:51:29.990183 13526 leader_election.cc:288] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225), d827d2628e984b3195cea1e7a4786634 (127.5.227.189:33863)
I20260928 10:51:29.992369 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.990456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51786)
0928 10:51:29.991860 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.992069 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1254,"thread_start_us":1144,"threads_started":1}
W20260928 10:51:29.992826 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:29.993064 13530 raft_consensus.cc:482] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbaae6cf9d0245659e76c2d34361d683)
I20260928 10:51:29.993297 13530 raft_consensus.cc:504] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } }
I20260928 10:51:29.994603 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.994140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51796)
0928 10:51:29.994334 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.994429 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 10:51:29.995056 13530 leader_election.cc:288] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225), 93894c132dae473d838d5f9bf5b3fba4 (127.5.227.188:34513)
W20260928 10:51:29.995297 12972 leader_election.cc:334] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225): Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:29.995550 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.994466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51804)
0928 10:51:29.995035 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.995266 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
W20260928 10:51:29.996568 12778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:29.998734 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.998046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51818)
0928 10:51:29.998216 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.998502 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 10:51:29.999004 12778 leader_election.cc:334] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225): Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:29.999358 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.990423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:33863 (local address 127.0.0.1:52032)
0928 10:51:29.991175 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.991202 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:29.991356 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:29.991921 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:29.991926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:29.991981 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:29.992309 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:29.992324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:29.994050 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:29.994056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:29.996459 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:29.996475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:29.998467 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:29.998475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:29.998686 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:29.998737 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 10:51:29.998842 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 10:51:29.998983 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":386,"threads_started":1}
I20260928 10:51:30.000061 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.990581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.5.227.189:33863)
0928 10:51:29.991543 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:29.991548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:29.991564 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:51:29.991624 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:29.991628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:29.991686 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:29.991877 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:29.992521 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:29.993805 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:29.996613 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:29.998259 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:29.998472 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:29.998585 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:29.998943 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 10:51:29.999133 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":549,"threads_started":1}
I20260928 10:51:30.000763 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93894c132dae473d838d5f9bf5b3fba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d827d2628e984b3195cea1e7a4786634" is_pre_election: true
I20260928 10:51:30.001081 12804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93894c132dae473d838d5f9bf5b3fba4 in term 1.
I20260928 10:51:30.001853 12974 leader_election.cc:302] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93894c132dae473d838d5f9bf5b3fba4, d827d2628e984b3195cea1e7a4786634; no voters: bbaae6cf9d0245659e76c2d34361d683
I20260928 10:51:30.002336 13526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 10:51:30.002631 13526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbaae6cf9d0245659e76c2d34361d683)
I20260928 10:51:30.002791 13526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:51:30.004714 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.995290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:34513 (local address 127.0.0.1:40562)
0928 10:51:29.995886 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:29.995907 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:29.996030 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:29.996721 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:29.996727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:29.996764 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:29.997149 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:29.997222 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:29.998861 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:29.998870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.001775 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.001791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.004032 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.004038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.004190 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.004223 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 10:51:30.004320 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 10:51:30.004485 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1,"thread_start_us":931,"threads_started":1}
I20260928 10:51:30.005091 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:29.995722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.5.227.188:34513)
0928 10:51:29.996349 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:29.996354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:29.996381 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:51:29.996445 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:29.996450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:29.996516 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:29.996678 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:29.997408 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:29.998730 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.001963 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.003773 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.003960 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.004105 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.004399 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.004648 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":314,"threads_started":1}
I20260928 10:51:30.006143 12999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d827d2628e984b3195cea1e7a4786634" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93894c132dae473d838d5f9bf5b3fba4" is_pre_election: true
I20260928 10:51:30.006973 13526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } }
I20260928 10:51:30.007555 12999 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d827d2628e984b3195cea1e7a4786634 in current term 2: Already voted for candidate 93894c132dae473d838d5f9bf5b3fba4 in this term.
I20260928 10:51:30.007916 13526 leader_election.cc:288] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [CANDIDATE]: Term 2 election: Requested vote from peers bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225), d827d2628e984b3195cea1e7a4786634 (127.5.227.189:33863)
I20260928 10:51:30.008343 12778 leader_election.cc:302] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d827d2628e984b3195cea1e7a4786634; no voters: 93894c132dae473d838d5f9bf5b3fba4, bbaae6cf9d0245659e76c2d34361d683
I20260928 10:51:30.008635 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.008050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51830)
0928 10:51:30.008307 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.008397 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 10:51:30.008664 13530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:51:30.008939 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93894c132dae473d838d5f9bf5b3fba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d827d2628e984b3195cea1e7a4786634"
I20260928 10:51:30.009865 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.009344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51842)
0928 10:51:30.009524 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.009747 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 10:51:30.010028 12972 leader_election.cc:334] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225): Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
I20260928 10:51:30.011466 13530 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 10:51:30.014125 12804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93894c132dae473d838d5f9bf5b3fba4 in term 2.
I20260928 10:51:30.015164 12974 leader_election.cc:302] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93894c132dae473d838d5f9bf5b3fba4, d827d2628e984b3195cea1e7a4786634; no voters: bbaae6cf9d0245659e76c2d34361d683
I20260928 10:51:30.015538 13526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:51:30.016045 13526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [term 2 LEADER]: Becoming Leader. State: Replica: 93894c132dae473d838d5f9bf5b3fba4, State: Running, Role: LEADER
I20260928 10:51:30.016541 13526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } }
I20260928 10:51:30.018954 13536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93894c132dae473d838d5f9bf5b3fba4. Latest consensus state: current_term: 2 leader_uuid: "93894c132dae473d838d5f9bf5b3fba4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:30.019469 13536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:30.020344 13538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:30.025286 13538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:30.026393 13538 catalog_manager.cc:1259] Loaded cluster ID: ecb34afe09b145fcb8491764a29d4a70
I20260928 10:51:30.026492 13538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:30.027890 13538 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:30.028803 13538 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Loaded TSK: 0
I20260928 10:51:30.029806 13538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:51:30.043156 12988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:51:30.050874 12988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:30.055828 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.055194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51864)
0928 10:51:30.055377 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.055582 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 10:51:30.055995 12804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [term 2 FOLLOWER]: Refusing update from remote peer 93894c132dae473d838d5f9bf5b3fba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 10:51:30.057240 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.056700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:51878)
0928 10:51:30.056973 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.057079 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 10:51:30.057154 13536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 10:51:30.057495 12972 consensus_peers.cc:487] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 -> Peer bbaae6cf9d0245659e76c2d34361d683 (127.5.227.190:36225): Couldn't send request to peer bbaae6cf9d0245659e76c2d34361d683. Status: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 10:51:30.060639 13530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93894c132dae473d838d5f9bf5b3fba4. Latest consensus state: current_term: 2 leader_uuid: "93894c132dae473d838d5f9bf5b3fba4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:30.060987 13530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:51:30.062942 13536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "93894c132dae473d838d5f9bf5b3fba4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:30.063345 13536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:30.064073 13530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "93894c132dae473d838d5f9bf5b3fba4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbaae6cf9d0245659e76c2d34361d683" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36225 } } peers { permanent_uuid: "d827d2628e984b3195cea1e7a4786634" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 33863 } attrs { promote: false } } peers { permanent_uuid: "93894c132dae473d838d5f9bf5b3fba4" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 34513 } attrs { promote: false } } }
I20260928 10:51:30.064435 13530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d827d2628e984b3195cea1e7a4786634 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:51:30.083304 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.073500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:45421 (local address 127.0.0.1:53618)
0928 10:51:30.074484 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.074507 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:30.074659 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:30.076534 (+ 1875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:30.076540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:30.076568 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:30.076918 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.076930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.078042 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.078050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.080489 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.080506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.082279 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.082286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.082589 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.082621 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:51:30.082770 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 10:51:30.082946 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837}
I20260928 10:51:30.084158 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.074954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.5.227.130:45421)
0928 10:51:30.076132 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.076137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.076155 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:51:30.076221 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.076224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.076284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.076508 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:30.077084 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.077881 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.080743 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.082014 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.082194 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.082317 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.082812 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.082978 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":964,"threads_started":1}
I20260928 10:51:30.085389 13248 tablet_service.cc:1467] Processing CreateTablet for tablet fd0bbe36e8194f22b844554fc045a19d (DEFAULT_TABLE table=test-table [id=662700d9ac7c4ff3bccd7b08900a8af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:30.085925 13248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0bbe36e8194f22b844554fc045a19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:30.092146 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.073134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:33795 (local address 127.0.0.1:58918)
0928 10:51:30.073301 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.073370 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:30.073482 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:30.086851 (+ 13369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:30.086857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:30.086894 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:30.087047 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.087058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.088353 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.088358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.089642 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.089656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.091202 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.091207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.091402 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.091436 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:51:30.091560 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 10:51:30.091818 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 10:51:30.092931 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.073979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37621 (local address 127.0.0.1:43064)
0928 10:51:30.074650 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.074667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:30.074765 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:30.085829 (+ 11064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:30.085835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:30.085867 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:30.086248 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.086263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.088039 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.088044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.090214 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.090231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.092037 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.092044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.092231 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.092270 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:51:30.092343 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 10:51:30.092654 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":1109,"threads_started":1}
I20260928 10:51:30.093163 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.073291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.5.227.131:33795)
0928 10:51:30.086313 (+ 13022us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.086318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.086333 (+ 15us) server_negotiation.cc:407] Connection header received
0928 10:51:30.086587 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.086593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.086683 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.086808 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:30.087214 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.088211 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.089842 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.091044 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.091201 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.091317 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.092632 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.092781 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12913,"thread_start_us":12798,"threads_started":1}
I20260928 10:51:30.094064 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.074122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.5.227.129:37621)
0928 10:51:30.085188 (+ 11066us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.085197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.085228 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:51:30.085326 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.085334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.085471 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.085776 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:30.086572 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.087882 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.090387 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.091765 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.093367 (+ 1602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.093513 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.093582 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.093742 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10886,"thread_start_us":11467,"threads_started":1}
I20260928 10:51:30.094183 13389 tablet_service.cc:1467] Processing CreateTablet for tablet fd0bbe36e8194f22b844554fc045a19d (DEFAULT_TABLE table=test-table [id=662700d9ac7c4ff3bccd7b08900a8af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:30.094847 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0bbe36e8194f22b844554fc045a19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:30.094897 13108 tablet_service.cc:1467] Processing CreateTablet for tablet fd0bbe36e8194f22b844554fc045a19d (DEFAULT_TABLE table=test-table [id=662700d9ac7c4ff3bccd7b08900a8af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:30.095407 13108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0bbe36e8194f22b844554fc045a19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:30.095997 13549 tablet_bootstrap.cc:492] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: Bootstrap starting.
I20260928 10:51:30.098199 13549 tablet_bootstrap.cc:654] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:30.103881 13551 tablet_bootstrap.cc:492] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: Bootstrap starting.
I20260928 10:51:30.107568 13552 tablet_bootstrap.cc:492] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: Bootstrap starting.
I20260928 10:51:30.107784 13551 tablet_bootstrap.cc:654] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:30.109720 13552 tablet_bootstrap.cc:654] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:30.111102 13551 tablet_bootstrap.cc:492] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: No bootstrap required, opened a new log
I20260928 10:51:30.111299 13551 ts_tablet_manager.cc:1397] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 10:51:30.111449 13549 tablet_bootstrap.cc:492] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: No bootstrap required, opened a new log
I20260928 10:51:30.111685 13549 ts_tablet_manager.cc:1397] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20260928 10:51:30.112491 13552 tablet_bootstrap.cc:492] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: No bootstrap required, opened a new log
I20260928 10:51:30.112316 13551 raft_consensus.cc:348] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.112605 13551 raft_consensus.cc:374] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:30.112659 13552 ts_tablet_manager.cc:1397] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 10:51:30.112712 13551 raft_consensus.cc:729] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569f926721f94d0d9d6102ae8eba1011, State: Initialized, Role: FOLLOWER
I20260928 10:51:30.112880 13549 raft_consensus.cc:348] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.113027 13551 consensus_queue.cc:260] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.113171 13549 raft_consensus.cc:374] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:30.113283 13549 raft_consensus.cc:729] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdf8c9f9d7d484a9f8fb96bfb632148, State: Initialized, Role: FOLLOWER
I20260928 10:51:30.113613 13552 raft_consensus.cc:348] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.113889 13552 raft_consensus.cc:374] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:30.113734 13549 consensus_queue.cc:260] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.113991 13552 raft_consensus.cc:729] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6458de1cef7a42e6b3344e806939474d, State: Initialized, Role: FOLLOWER
I20260928 10:51:30.114288 13551 ts_tablet_manager.cc:1428] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 10:51:30.114243 13552 consensus_queue.cc:260] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.117703 13552 ts_tablet_manager.cc:1428] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260928 10:51:30.120622 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.117982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.131:54057)
0928 10:51:30.120239 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.120399 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2098,"thread_start_us":1948,"threads_started":1}
I20260928 10:51:30.125051 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.121353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.131:51629)
0928 10:51:30.124731 (+ 3378us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.124839 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3242,"threads_started":1}
I20260928 10:51:30.130694 13549 ts_tablet_manager.cc:1428] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: Time spent starting tablet: real 0.019s user 0.006s sys 0.012s
I20260928 10:51:30.137404 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.116420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.129:40571)
0928 10:51:30.136791 (+ 20371us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.137022 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19921,"threads_started":1}
I20260928 10:51:30.138873 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.131702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.130:59285)
0928 10:51:30.138283 (+ 6581us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.138598 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6431,"thread_start_us":6286,"threads_started":1}
I20260928 10:51:30.140682 13455 heartbeater.cc:503] Master 127.5.227.188:34513 was elected leader, sending a full tablet report...
I20260928 10:51:30.140780 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.140053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.130:52801)
0928 10:51:30.140312 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.140604 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 10:51:30.141793 13314 heartbeater.cc:503] Master 127.5.227.188:34513 was elected leader, sending a full tablet report...
I20260928 10:51:30.250013 13555 raft_consensus.cc:482] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:30.250283 13555 raft_consensus.cc:504] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.252004 13555 leader_election.cc:288] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6458de1cef7a42e6b3344e806939474d (127.5.227.131:33795), fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421)
I20260928 10:51:30.252696 13557 raft_consensus.cc:482] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:30.252939 13557 raft_consensus.cc:504] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.253921 13557 leader_election.cc:288] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421), 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621)
I20260928 10:51:30.254640 13128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0bbe36e8194f22b844554fc045a19d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569f926721f94d0d9d6102ae8eba1011" is_pre_election: true
I20260928 10:51:30.254634 13268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0bbe36e8194f22b844554fc045a19d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" is_pre_election: true
I20260928 10:51:30.254889 13128 raft_consensus.cc:2454] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 0.
I20260928 10:51:30.254890 13268 raft_consensus.cc:2454] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 0.
I20260928 10:51:30.255385 13343 leader_election.cc:302] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569f926721f94d0d9d6102ae8eba1011, 6458de1cef7a42e6b3344e806939474d; no voters:
I20260928 10:51:30.255801 13557 raft_consensus.cc:2781] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:30.255937 13557 raft_consensus.cc:482] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:30.256031 13557 raft_consensus.cc:3037] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:30.259064 13557 raft_consensus.cc:504] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.259935 13557 leader_election.cc:288] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 election: Requested vote from peers fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421), 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621)
I20260928 10:51:30.260593 13268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0bbe36e8194f22b844554fc045a19d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148"
I20260928 10:51:30.260634 13128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0bbe36e8194f22b844554fc045a19d" candidate_uuid: "6458de1cef7a42e6b3344e806939474d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569f926721f94d0d9d6102ae8eba1011"
I20260928 10:51:30.260842 13128 raft_consensus.cc:3037] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:30.260838 13268 raft_consensus.cc:3037] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:30.261705 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.253230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:40233 (local address 127.5.227.130:45421)
0928 10:51:30.253548 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.253554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.254310 (+ 756us) server_negotiation.cc:407] Connection header received
0928 10:51:30.254566 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.254572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.254652 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.254720 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:30.255678 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.256610 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.259064 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.260338 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.260535 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.260623 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.261227 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.261389 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260928 10:51:30.261830 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.253004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:45421 (local address 127.5.227.129:40233)
0928 10:51:30.254209 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.254228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:30.254403 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:30.255082 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:30.255086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:30.255112 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:30.255454 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.255466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.256779 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.256785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.258782 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.258800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.260782 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.260789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.261002 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.261056 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 10:51:30.261166 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 10:51:30.261331 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":64,"thread_start_us":759,"threads_started":1}
I20260928 10:51:30.263121 13267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0bbe36e8194f22b844554fc045a19d" candidate_uuid: "569f926721f94d0d9d6102ae8eba1011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" is_pre_election: true
I20260928 10:51:30.263772 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.252441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:58923 (local address 127.5.227.131:33795)
0928 10:51:30.252638 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.252643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.252978 (+ 335us) server_negotiation.cc:407] Connection header received
0928 10:51:30.253302 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.253309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.253407 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.253490 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:30.254403 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.255481 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.258490 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.260102 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.260262 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.260405 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.263101 (+ 2696us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.263334 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 10:51:30.264149 13268 raft_consensus.cc:2454] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 1.
I20260928 10:51:30.264571 13128 raft_consensus.cc:2454] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6458de1cef7a42e6b3344e806939474d in term 1.
I20260928 10:51:30.265110 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.252512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:33795 (local address 127.5.227.129:58923)
0928 10:51:30.252775 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.252798 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:30.253103 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:30.253775 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:30.253781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:30.253822 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:30.254211 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.254226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.255652 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.255658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.258197 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:30.258215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.262589 (+ 4374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.262596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:30.262835 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.262877 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 10:51:30.264083 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0928 10:51:30.264248 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":35}
I20260928 10:51:30.265110 13343 leader_election.cc:302] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6458de1cef7a42e6b3344e806939474d, fcdf8c9f9d7d484a9f8fb96bfb632148; no voters:
I20260928 10:51:30.265818 13557 raft_consensus.cc:2781] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:30.266609 13557 raft_consensus.cc:686] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 1 LEADER]: Becoming Leader. State: Replica: 6458de1cef7a42e6b3344e806939474d, State: Running, Role: LEADER
I20260928 10:51:30.266610 13409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0bbe36e8194f22b844554fc045a19d" candidate_uuid: "569f926721f94d0d9d6102ae8eba1011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6458de1cef7a42e6b3344e806939474d" is_pre_election: true
I20260928 10:51:30.267401 13557 consensus_queue.cc:237] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } }
I20260928 10:51:30.268185 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.267225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.131:57629)
0928 10:51:30.267533 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.267866 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 10:51:30.268209 13062 leader_election.cc:302] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 569f926721f94d0d9d6102ae8eba1011; no voters: 6458de1cef7a42e6b3344e806939474d, fcdf8c9f9d7d484a9f8fb96bfb632148
I20260928 10:51:30.268659 13555 raft_consensus.cc:2726] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 10:51:30.273626 12988 catalog_manager.cc:5495] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6458de1cef7a42e6b3344e806939474d (127.5.227.131). New cstate: current_term: 1 leader_uuid: "6458de1cef7a42e6b3344e806939474d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6458de1cef7a42e6b3344e806939474d" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 33795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 } health_report { overall_health: UNKNOWN } } }
I20260928 10:51:30.334537 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.328757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.5.227.131:33795)
0928 10:51:30.329009 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.329017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.329185 (+ 168us) server_negotiation.cc:407] Connection header received
0928 10:51:30.329518 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.329523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.329595 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.329706 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:30.330124 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.330966 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.332456 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.332949 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.333105 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.333544 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:30.333670 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:30.333783 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.334021 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.334107 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 10:51:30.338165 13128 raft_consensus.cc:1261] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Refusing update from remote peer 6458de1cef7a42e6b3344e806939474d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:30.338222 13268 raft_consensus.cc:1261] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Refusing update from remote peer 6458de1cef7a42e6b3344e806939474d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:30.338907 13563 consensus_queue.cc:1035] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [LEADER]: Connected to new peer: Peer: permanent_uuid: "569f926721f94d0d9d6102ae8eba1011" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:30.339318 13557 consensus_queue.cc:1035] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcdf8c9f9d7d484a9f8fb96bfb632148" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:30.342244 13174 heartbeater.cc:503] Master 127.5.227.188:34513 was elected leader, sending a full tablet report...
I20260928 10:51:30.342645 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.341715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.129:38541)
0928 10:51:30.342127 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.342277 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250}
I20260928 10:51:30.344769 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.344070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.130:54247)
0928 10:51:30.344335 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.344537 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 10:51:30.353499 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.352969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.130:40053)
0928 10:51:30.353206 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.353309 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 10:51:30.361059 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.360499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.129:60943)
0928 10:51:30.360752 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.360864 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 10:51:30.369904 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.368968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.5.227.131:59215)
0928 10:51:30.369518 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.369656 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383}
I20260928 10:51:30.379349 12989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:30.379927 12989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:30.388484 12989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93894c132dae473d838d5f9bf5b3fba4: Sending DeleteTablet for 3 replicas of tablet fd0bbe36e8194f22b844554fc045a19d
I20260928 10:51:30.389845 13389 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0bbe36e8194f22b844554fc045a19d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:30 UTC) from {username='oryx', principal='oryx/127.5.227.188@KRBTEST.COM'} at 127.0.0.1:58918
I20260928 10:51:30.390194 13108 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0bbe36e8194f22b844554fc045a19d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:30 UTC) from {username='oryx', principal='oryx/127.5.227.188@KRBTEST.COM'} at 127.0.0.1:43064
I20260928 10:51:30.390746 13248 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0bbe36e8194f22b844554fc045a19d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:30 UTC) from {username='oryx', principal='oryx/127.5.227.188@KRBTEST.COM'} at 127.0.0.1:53618
I20260928 10:51:30.391124 13575 tablet_replica.cc:330] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: stopping tablet replica
I20260928 10:51:30.391909 13576 tablet_replica.cc:330] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: stopping tablet replica
I20260928 10:51:30.392270 13576 raft_consensus.cc:2229] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:30.392795 13576 raft_consensus.cc:2258] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:30.393464 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 12976
I20260928 10:51:30.394749 13576 ts_tablet_manager.cc:1905] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:30.396241 13575 raft_consensus.cc:2229] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:30.396677 13575 raft_consensus.cc:2258] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:30.397984 13575 ts_tablet_manager.cc:1905] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:30.402508 13574 tablet_replica.cc:330] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: stopping tablet replica
I20260928 10:51:30.402772 13574 raft_consensus.cc:2229] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:30.403303 13574 raft_consensus.cc:2258] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:30.404932 13574 ts_tablet_manager.cc:1905] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:30.408726 13575 ts_tablet_manager.cc:1918] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:30.408990 13575 log.cc:1196] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/fd0bbe36e8194f22b844554fc045a19d
I20260928 10:51:30.410171 13575 ts_tablet_manager.cc:1939] T fd0bbe36e8194f22b844554fc045a19d P 569f926721f94d0d9d6102ae8eba1011: Deleting consensus metadata
I20260928 10:51:30.412678 13574 ts_tablet_manager.cc:1918] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:30.412842 13574 log.cc:1196] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/fd0bbe36e8194f22b844554fc045a19d
I20260928 10:51:30.412690 12973 catalog_manager.cc:4841] TS 569f926721f94d0d9d6102ae8eba1011 (127.5.227.129:37621): tablet fd0bbe36e8194f22b844554fc045a19d (table test-table [id=662700d9ac7c4ff3bccd7b08900a8af9]) successfully deleted
I20260928 10:51:30.412700 13576 ts_tablet_manager.cc:1918] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:30.413229 13576 log.cc:1196] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/fd0bbe36e8194f22b844554fc045a19d
I20260928 10:51:30.413332 13574 ts_tablet_manager.cc:1939] T fd0bbe36e8194f22b844554fc045a19d P 6458de1cef7a42e6b3344e806939474d: Deleting consensus metadata
I20260928 10:51:30.413854 13576 ts_tablet_manager.cc:1939] T fd0bbe36e8194f22b844554fc045a19d P fcdf8c9f9d7d484a9f8fb96bfb632148: Deleting consensus metadata
I20260928 10:51:30.414716 12973 catalog_manager.cc:4841] TS 6458de1cef7a42e6b3344e806939474d (127.5.227.131:33795): tablet fd0bbe36e8194f22b844554fc045a19d (table test-table [id=662700d9ac7c4ff3bccd7b08900a8af9]) successfully deleted
I20260928 10:51:30.415126 12973 catalog_manager.cc:4841] TS fcdf8c9f9d7d484a9f8fb96bfb632148 (127.5.227.130:45421): tablet fd0bbe36e8194f22b844554fc045a19d (table test-table [id=662700d9ac7c4ff3bccd7b08900a8af9]) successfully deleted
I20260928 10:51:30.429682 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.395655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.5.227.129:37621)
0928 10:51:30.396004 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.396014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.405163 (+ 9149us) server_negotiation.cc:407] Connection header received
0928 10:51:30.405603 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.405608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.405700 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.405824 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:30.407845 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.409146 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.414021 (+ 4875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.415379 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.415588 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.418827 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:30.418852 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:30.418884 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:30.418949 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:30.424845 (+ 5896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:30.426136 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:30.426153 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:30.426159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:30.426659 (+ 500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:30.427147 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:30.427155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:30.427160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:30.427951 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:30.428405 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.428667 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.429062 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 10:51:30.430528 13168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:30.535207 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 12976
I20260928 10:51:30.546010 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.545232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:46644)
0928 10:51:30.545525 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.545790 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260928 10:51:30.547322 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.546923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36225 (local address 127.0.0.1:46648)
0928 10:51:30.547087 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:30.547180 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 10:51:30.556023 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 13185
I20260928 10:51:30.573396 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.556947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.5.227.130:45421)
0928 10:51:30.557174 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.557179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.557198 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:51:30.557512 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.557517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.557590 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.557694 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:30.559079 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.560073 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.561489 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.561982 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.562101 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.565765 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:30.565785 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:30.565802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:30.565847 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:30.569543 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:30.570624 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:30.570633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:30.570636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:30.570789 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:30.571282 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:30.571285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:30.571287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:30.571684 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:30.572112 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.572696 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.573045 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:51:30.574252 13308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:30.685592 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 13185
I20260928 10:51:30.709818 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 13326
I20260928 10:51:30.729254 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:30.710700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.5.227.131:33795)
0928 10:51:30.710950 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:30.710956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:30.710978 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:51:30.711081 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:30.711090 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:30.711167 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:30.711278 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:30.712840 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.713627 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.714985 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:30.715489 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:30.715641 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:30.719232 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:30.719249 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:30.719267 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:30.719324 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:30.723720 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:30.725184 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:30.725202 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:30.725206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:30.725667 (+ 461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:30.726559 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:30.726568 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:30.726571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:30.727208 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:30.728069 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:30.728605 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 10:51:30.728820 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 10:51:30.730278 13449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:30.873579 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 13326
I20260928 10:51:30.902120 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 12650
I20260928 10:51:30.904031 12824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:31.044983 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 12650
I20260928 10:51:31.069897 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 12758
I20260928 10:51:31.071221 13019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:31.209331 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 12758
2026-09-28T10:51:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11094 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:51:31 dist-test-slave-8w8g krb5kdc[13586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:51:31 dist-test-slave-8w8g krb5kdc[13586](info): set up 2 sockets
Sep 28 10:51:31 dist-test-slave-8w8g krb5kdc[13586](info): commencing operation
krb5kdc: starting...
W20260928 10:51:32.465830 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:51:32 dist-test-slave-8w8g krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab.
Entry for principal oryx/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab.
I20260928 10:51:32.767136 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40249
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab
--principal=oryx/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:40249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:33.175945 13599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:33.176303 13599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:33.176406 13599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:33.185495 13599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:33.185635 13599 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:51:33.185678 13599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:33.185732 13599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:33.185775 13599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:33.185830 13599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:33.199211 13599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:40249
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.190.keytab
--principal=oryx/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:33.201803 13599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:33.204241 13599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:33.224582 13605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:33.224611 13604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:33.224656 13607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:33.225492 13599 server_base.cc:1034] running on GCE node
I20260928 10:51:33.226873 13599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:33.229104 13599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:33.230566 13599 hybrid_clock.cc:648] HybridClock initialized: now 1790592693230435 us; error 103 us; skew 500 ppm
I20260928 10:51:33.231040 13599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:33 dist-test-slave-8w8g krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592693, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:33.245927 13599 init.cc:377] Logged in from keytab as oryx/127.5.227.190@KRBTEST.COM (short username oryx)
I20260928 10:51:33.248670 13599 webserver.cc:460] Webserver started at http://127.5.227.190:43657/ using document root <none> and password file <none>
I20260928 10:51:33.249642 13599 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:33.249768 13599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:33.250211 13599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:33.254465 13599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "fc62d2466097448abf283907400e0b6b"
format_stamp: "Formatted at 2026-09-28 10:51:33 on dist-test-slave-8w8g"
server_key: "43f59b8509665c97903b4b29909f654e"
server_key_iv: "9485b11ef4330a493665a2fbdc3b4f4e"
server_key_version: "encryptionkey@0"
I20260928 10:51:33.255891 13599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "fc62d2466097448abf283907400e0b6b"
format_stamp: "Formatted at 2026-09-28 10:51:33 on dist-test-slave-8w8g"
server_key: "43f59b8509665c97903b4b29909f654e"
server_key_iv: "9485b11ef4330a493665a2fbdc3b4f4e"
server_key_version: "encryptionkey@0"
I20260928 10:51:33.264087 13599 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260928 10:51:33.269464 13614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:33.271693 13599 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:51:33.272044 13599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "fc62d2466097448abf283907400e0b6b"
format_stamp: "Formatted at 2026-09-28 10:51:33 on dist-test-slave-8w8g"
server_key: "43f59b8509665c97903b4b29909f654e"
server_key_iv: "9485b11ef4330a493665a2fbdc3b4f4e"
server_key_version: "encryptionkey@0"
I20260928 10:51:33.272542 13599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:33.302702 13599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:33.309074 13599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:33.309417 13599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:33.340197 13599 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:40249
I20260928 10:51:33.340313 13675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:40249 every 8 connection(s)
I20260928 10:51:33.341756 13599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:33.348788 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 13599
I20260928 10:51:33.348830 13676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:33.349058 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:33.349960 6030 external_mini_cluster.cc:1428] Setting key 69dfb1af234c76bdba116103bab54f64
I20260928 10:51:33.364943 13676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Bootstrap starting.
Sep 28 10:51:33 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:33.369582 13676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:33.371481 13676 log.cc:824] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:33.376219 13676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: No bootstrap required, opened a new log
I20260928 10:51:33.380487 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:33.352962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.5.227.190:40249)
0928 10:51:33.353904 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:33.353926 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:33.353981 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:51:33.356415 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:33.356449 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:33.357013 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:33.357674 (+ 661us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:33.359306 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:33.360858 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:33.362280 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:33.363160 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:33.363398 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:33.368908 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:33.368919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:33.368939 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:33.368989 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:33.373715 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:33.375080 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:33.375088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:33.375096 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:33.375347 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:33.376097 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:33.376104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:33.376107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:33.377104 (+ 997us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:33.377514 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:33.378578 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0928 10:51:33.379140 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":470,"threads_started":1}
I20260928 10:51:33.389496 13676 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc62d2466097448abf283907400e0b6b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40249 } }
I20260928 10:51:33.390002 13676 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:33.390136 13676 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc62d2466097448abf283907400e0b6b, State: Initialized, Role: FOLLOWER
I20260928 10:51:33.390875 13676 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc62d2466097448abf283907400e0b6b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40249 } }
I20260928 10:51:33.391172 13676 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:33.391335 13676 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:33.391564 13676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:33.394462 13676 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc62d2466097448abf283907400e0b6b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40249 } }
I20260928 10:51:33.395090 13676 leader_election.cc:302] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc62d2466097448abf283907400e0b6b; no voters:
I20260928 10:51:33.395911 13676 leader_election.cc:288] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:33.396082 13681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:33.397259 13681 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [term 1 LEADER]: Becoming Leader. State: Replica: fc62d2466097448abf283907400e0b6b, State: Running, Role: LEADER
I20260928 10:51:33.398123 13681 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc62d2466097448abf283907400e0b6b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40249 } }
I20260928 10:51:33.399315 13676 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:33.402786 13683 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc62d2466097448abf283907400e0b6b. Latest consensus state: current_term: 1 leader_uuid: "fc62d2466097448abf283907400e0b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc62d2466097448abf283907400e0b6b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40249 } } }
I20260928 10:51:33.403386 13683 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:33.404423 13682 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc62d2466097448abf283907400e0b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc62d2466097448abf283907400e0b6b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 40249 } } }
I20260928 10:51:33.405062 13682 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:33.406867 13690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:33.412710 13690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:33.424739 13690 catalog_manager.cc:1347] Generated new cluster ID: d4db2031e30749c1bde652be6c4f126e
I20260928 10:51:33.424889 13690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:33.432736 13690 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:33.434613 13690 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:33.448941 13690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Generated new TSK 0
I20260928 10:51:33.450258 13690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab.
Entry for principal oryx/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab.
I20260928 10:51:33.534101 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40249
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab
--principal=oryx/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:33.951261 13702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:33.951663 13702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:33.951864 13702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:33.960189 13702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:33.960425 13702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:33.960589 13702 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:51:33.960733 13702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:33.971390 13702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab
--principal=oryx/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:40249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:33.972885 13702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:33.974339 13702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:33.989682 13707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:33.990341 13710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:33.990352 13708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:33.991119 13702 server_base.cc:1034] running on GCE node
I20260928 10:51:33.991943 13702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:33.993094 13702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:33.994393 13702 hybrid_clock.cc:648] HybridClock initialized: now 1790592693994352 us; error 57 us; skew 500 ppm
I20260928 10:51:33.994798 13702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:33 dist-test-slave-8w8g krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592693, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:33.998672 13702 init.cc:377] Logged in from keytab as oryx/127.5.227.129@KRBTEST.COM (short username oryx)
I20260928 10:51:34.000908 13702 webserver.cc:460] Webserver started at http://127.5.227.129:38677/ using document root <none> and password file <none>
I20260928 10:51:34.001961 13702 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:34.002079 13702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:34.002512 13702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:34.006202 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "aae97524b417474c970c9daf1b3b0eda"
format_stamp: "Formatted at 2026-09-28 10:51:33 on dist-test-slave-8w8g"
server_key: "a6ad7d2bc9eadd6c149f2767f0baf5e7"
server_key_iv: "acae81fb6f64e71ce47ad7fce3556503"
server_key_version: "encryptionkey@0"
I20260928 10:51:34.007125 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "aae97524b417474c970c9daf1b3b0eda"
format_stamp: "Formatted at 2026-09-28 10:51:33 on dist-test-slave-8w8g"
server_key: "a6ad7d2bc9eadd6c149f2767f0baf5e7"
server_key_iv: "acae81fb6f64e71ce47ad7fce3556503"
server_key_version: "encryptionkey@0"
I20260928 10:51:34.013569 13702 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260928 10:51:34.018265 13717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:34.020493 13702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:51:34.020715 13702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "aae97524b417474c970c9daf1b3b0eda"
format_stamp: "Formatted at 2026-09-28 10:51:33 on dist-test-slave-8w8g"
server_key: "a6ad7d2bc9eadd6c149f2767f0baf5e7"
server_key_iv: "acae81fb6f64e71ce47ad7fce3556503"
server_key_version: "encryptionkey@0"
I20260928 10:51:34.020995 13702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:34.044319 13702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:34.049885 13702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:34.050216 13702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:34.053755 13702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:34.053892 13702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:34.054045 13702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:34.054108 13702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:51:34 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592693, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:34.083302 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.055253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:33111 (local address 127.5.227.190:40249)
0928 10:51:34.055770 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:34.055777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:34.057199 (+ 1422us) server_negotiation.cc:407] Connection header received
0928 10:51:34.060818 (+ 3619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:34.060826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:34.060954 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:34.061111 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:34.063211 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.064021 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.068067 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.068571 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.068780 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.075207 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:34.075213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:34.075216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:34.075261 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:34.079117 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.080553 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.080569 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.080575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.080737 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.081413 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.081417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.081419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.081706 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:34.081872 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:34.082676 (+ 804us) server_negotiation.cc:299] Negotiation successful
0928 10:51:34.082922 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260928 10:51:34.084854 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.055989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40249 (local address 127.5.227.129:33111)
0928 10:51:34.056959 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:34.057024 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:34.060353 (+ 3329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:34.061496 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:34.061516 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:34.062299 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:34.062965 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.063011 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.064266 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.064276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.067813 (+ 3537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.067839 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.069149 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.069156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.069563 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.070433 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:34.070521 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:34.074843 (+ 4322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:34.079354 (+ 4511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.079372 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.079403 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.080304 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.080870 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.080879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.080883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.081229 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.081849 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:34.081858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:34.082279 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 10:51:34.082572 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 10:51:34.083154 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":363,"threads_started":1}
I20260928 10:51:34.112571 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.091547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40249 (local address 127.5.227.129:59989)
0928 10:51:34.092566 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:34.092593 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:34.092824 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:34.093302 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:34.093309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:34.094171 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:34.095011 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.095033 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.096952 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.096959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.098539 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.098557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.099976 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.099989 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.100312 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.101407 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:34.101438 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:34.102801 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:34.107578 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.107590 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.107596 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.108633 (+ 1037us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.109357 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.109366 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.109372 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.109604 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.110407 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:34.110415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:34.110698 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 10:51:34.111225 (+ 527us) client_negotiation.cc:240] Negotiation successful
0928 10:51:34.112005 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":588,"threads_started":1}
I20260928 10:51:34.113755 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.091626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:59989 (local address 127.5.227.190:40249)
0928 10:51:34.092048 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:34.092059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:34.092779 (+ 720us) server_negotiation.cc:407] Connection header received
0928 10:51:34.093039 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:34.093054 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:34.093126 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:34.093223 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:34.095343 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.096777 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.098821 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.099602 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.099817 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.103102 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:34.103107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:34.103111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:34.103179 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:34.107340 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.108884 (+ 1544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.108893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.108899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.109152 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.109776 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.109781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.109783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.110260 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:34.110452 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:34.113056 (+ 2604us) server_negotiation.cc:299] Negotiation successful
0928 10:51:34.113277 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 10:51:34.124444 13702 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:37363
I20260928 10:51:34.124473 13837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:37363 every 8 connection(s)
I20260928 10:51:34.125881 13702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:34.127012 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 13702
I20260928 10:51:34.127230 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:51:34.128362 6030 external_mini_cluster.cc:1428] Setting key 8c875701e3c0f7463eb50d4dda90dfcd
I20260928 10:51:34.128537 13838 heartbeater.cc:347] Connected to a master server at 127.5.227.190:40249
I20260928 10:51:34.128790 13838 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:34.129637 13838 heartbeater.cc:511] Master 127.5.227.190:40249 requested a full tablet report, sending...
I20260928 10:51:34.132230 13631 ts_manager.cc:194] Registered new tserver with Master: aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363)
I20260928 10:51:34.134474 13631 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:33111
WARNING: no policy specified for oryx/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab.
Entry for principal oryx/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab.
I20260928 10:51:34.194522 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40249
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab
--principal=oryx/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:34.565400 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:34.565729 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:34.565973 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:34.573843 13842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:34.573989 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:34.574138 13842 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:51:34.574249 13842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:34.585791 13842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab
--principal=oryx/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:40249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:34.587419 13842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:34.589143 13842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:34.604225 13847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:34.604308 13850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:34.604329 13848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:34.606734 13842 server_base.cc:1034] running on GCE node
I20260928 10:51:34.607564 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:34.608757 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:34.610184 13842 hybrid_clock.cc:648] HybridClock initialized: now 1790592694610121 us; error 42 us; skew 500 ppm
I20260928 10:51:34.610508 13842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:34 dist-test-slave-8w8g krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592694, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:34.614377 13842 init.cc:377] Logged in from keytab as oryx/127.5.227.130@KRBTEST.COM (short username oryx)
I20260928 10:51:34.616103 13842 webserver.cc:460] Webserver started at http://127.5.227.130:33653/ using document root <none> and password file <none>
I20260928 10:51:34.616981 13842 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:34.617129 13842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:34.617736 13842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:34.621852 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "af3b45d7801c456da77fe8d6093af740"
format_stamp: "Formatted at 2026-09-28 10:51:34 on dist-test-slave-8w8g"
server_key: "b8916154f8210471dac387d08051545f"
server_key_iv: "31f3865e6d0ca1ebe936194fcbe3bca2"
server_key_version: "encryptionkey@0"
I20260928 10:51:34.623044 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "af3b45d7801c456da77fe8d6093af740"
format_stamp: "Formatted at 2026-09-28 10:51:34 on dist-test-slave-8w8g"
server_key: "b8916154f8210471dac387d08051545f"
server_key_iv: "31f3865e6d0ca1ebe936194fcbe3bca2"
server_key_version: "encryptionkey@0"
I20260928 10:51:34.630147 13842 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260928 10:51:34.635812 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:34.637709 13842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 10:51:34.637929 13842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "af3b45d7801c456da77fe8d6093af740"
format_stamp: "Formatted at 2026-09-28 10:51:34 on dist-test-slave-8w8g"
server_key: "b8916154f8210471dac387d08051545f"
server_key_iv: "31f3865e6d0ca1ebe936194fcbe3bca2"
server_key_version: "encryptionkey@0"
I20260928 10:51:34.638171 13842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:34.662971 13842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:34.669221 13842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:34.669657 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:34.673058 13842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:34.673183 13842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:34.673357 13842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:34.673450 13842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:51:34 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592694, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:34.704507 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.673678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:34357 (local address 127.5.227.190:40249)
0928 10:51:34.674257 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:34.674263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:34.675703 (+ 1440us) server_negotiation.cc:407] Connection header received
0928 10:51:34.679641 (+ 3938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:34.679647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:34.679787 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:34.679934 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:34.683118 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.684819 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.687840 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.688521 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.688686 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.694967 (+ 6281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:34.694976 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:34.694981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:34.695034 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:34.699521 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.700887 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.700896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.700900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.701092 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.701887 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.701892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.701896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.702330 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:34.702516 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:34.703668 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0928 10:51:34.704009 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":280,"threads_started":1}
I20260928 10:51:34.705713 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.674492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40249 (local address 127.5.227.130:34357)
0928 10:51:34.675467 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:34.675524 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:34.679199 (+ 3675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:34.680277 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:34.680295 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:34.681662 (+ 1367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:34.682805 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.682835 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.685108 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.685120 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.687540 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.687564 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.688853 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.688864 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.689181 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.690197 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:34.690243 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:34.694559 (+ 4316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:34.699805 (+ 5246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.699819 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.699846 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.700641 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.701239 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.701247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.701251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.701751 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.702620 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:34.702627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:34.703228 (+ 601us) client_negotiation.cc:769] Sending connection context
0928 10:51:34.703545 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 10:51:34.704236 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":426,"threads_started":1}
I20260928 10:51:34.741495 13842 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:44351
I20260928 10:51:34.741600 13977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:44351 every 8 connection(s)
I20260928 10:51:34.743134 13842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:34.744405 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 13842
I20260928 10:51:34.744654 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:51:34.745433 6030 external_mini_cluster.cc:1428] Setting key 92bb4b7ed20b2e5bf0e9adfaaa7b7e75
I20260928 10:51:34.747695 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.721568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:46705 (local address 127.5.227.190:40249)
0928 10:51:34.721863 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:34.721871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:34.724629 (+ 2758us) server_negotiation.cc:407] Connection header received
0928 10:51:34.724908 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:34.724921 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:34.725733 (+ 812us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:34.725886 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:34.728532 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.729746 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.733778 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.734689 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.734899 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.739952 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:34.739966 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:34.739974 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:34.740025 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:34.743941 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.745023 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.745031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.745035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.745183 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:34.745622 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:34.745632 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:34.745634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:34.745980 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:34.746154 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:34.747031 (+ 877us) server_negotiation.cc:299] Negotiation successful
0928 10:51:34.747314 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 10:51:34.747958 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:34.722218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40249 (local address 127.5.227.130:46705)
0928 10:51:34.724307 (+ 2089us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:34.724333 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:34.724718 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:34.725994 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:34.726006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:34.727372 (+ 1366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:34.728206 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.728229 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.729978 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.729997 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.733221 (+ 3224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:34.733254 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:34.735014 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:34.735028 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:34.735311 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:34.737786 (+ 2475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:34.737820 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:34.739277 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:34.744186 (+ 4909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.744198 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.744202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.744826 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.745353 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:34.745358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:34.745362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:34.745495 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:34.746115 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:34.746120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:34.746655 (+ 535us) client_negotiation.cc:769] Sending connection context
0928 10:51:34.746909 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 10:51:34.747310 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1867,"thread_start_us":1006,"threads_started":1}
I20260928 10:51:34.748920 13978 heartbeater.cc:347] Connected to a master server at 127.5.227.190:40249
I20260928 10:51:34.749369 13978 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:34.750212 13978 heartbeater.cc:511] Master 127.5.227.190:40249 requested a full tablet report, sending...
I20260928 10:51:34.752439 13631 ts_manager.cc:194] Registered new tserver with Master: af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351)
I20260928 10:51:34.753666 13631 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:34357
WARNING: no policy specified for oryx/127.5.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab.
Entry for principal oryx/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab.
I20260928 10:51:34.825366 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:40249
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab
--principal=oryx/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:35.138152 13838 heartbeater.cc:503] Master 127.5.227.190:40249 was elected leader, sending a full tablet report...
W20260928 10:51:35.231338 13982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:35.231678 13982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:35.231837 13982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:35.240172 13982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:35.240337 13982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:35.240442 13982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:51:35.240531 13982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:35.251470 13982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab
--principal=oryx/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.227.190:40249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:35.253062 13982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:35.254594 13982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:35.272259 13988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:35.272434 13990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:35.272604 13987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:35.273327 13982 server_base.cc:1034] running on GCE node
I20260928 10:51:35.274098 13982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:35.275271 13982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:35.276587 13982 hybrid_clock.cc:648] HybridClock initialized: now 1790592695276466 us; error 93 us; skew 500 ppm
I20260928 10:51:35.277074 13982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:35 dist-test-slave-8w8g krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592695, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:35.281755 13982 init.cc:377] Logged in from keytab as oryx/127.5.227.131@KRBTEST.COM (short username oryx)
I20260928 10:51:35.283614 13982 webserver.cc:460] Webserver started at http://127.5.227.131:39305/ using document root <none> and password file <none>
I20260928 10:51:35.284541 13982 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:35.284647 13982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:35.284945 13982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:35.288435 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "a9f83f11adea4c869cfb4cc27c4e563f"
format_stamp: "Formatted at 2026-09-28 10:51:35 on dist-test-slave-8w8g"
server_key: "2fabfd097f8d87d06dd933c677881e50"
server_key_iv: "481d30a10cd9d9340945ccd8e5672267"
server_key_version: "encryptionkey@0"
I20260928 10:51:35.289356 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "a9f83f11adea4c869cfb4cc27c4e563f"
format_stamp: "Formatted at 2026-09-28 10:51:35 on dist-test-slave-8w8g"
server_key: "2fabfd097f8d87d06dd933c677881e50"
server_key_iv: "481d30a10cd9d9340945ccd8e5672267"
server_key_version: "encryptionkey@0"
I20260928 10:51:35.295459 13982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:51:35.300290 13997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:35.302284 13982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:51:35.302479 13982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "a9f83f11adea4c869cfb4cc27c4e563f"
format_stamp: "Formatted at 2026-09-28 10:51:35 on dist-test-slave-8w8g"
server_key: "2fabfd097f8d87d06dd933c677881e50"
server_key_iv: "481d30a10cd9d9340945ccd8e5672267"
server_key_version: "encryptionkey@0"
I20260928 10:51:35.302704 13982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:35.319422 13982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:35.325361 13982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:35.325773 13982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:35.330018 13982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:35.330127 13982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:35.331468 13982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:35.331593 13982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
Sep 28 10:51:35 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592695, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:35.361433 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:35.330594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:36997 (local address 127.5.227.190:40249)
0928 10:51:35.331609 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:35.331617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:35.332459 (+ 842us) server_negotiation.cc:407] Connection header received
0928 10:51:35.335453 (+ 2994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:35.335460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:35.335574 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:35.335704 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:35.338129 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.339902 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.344480 (+ 4578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.345589 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.345860 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:35.350946 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:35.350951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:35.350955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:35.350993 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:35.356028 (+ 5035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:35.357592 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:35.357598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:35.357600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:35.357742 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:35.358466 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:35.358474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:35.358483 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:35.358839 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:35.359050 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:35.360193 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0928 10:51:35.360721 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":587,"threads_started":1}
I20260928 10:51:35.362996 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:35.331167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40249 (local address 127.5.227.131:36997)
0928 10:51:35.332279 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:35.332330 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:35.335052 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:35.336035 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:35.336045 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:35.336662 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:35.337861 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:35.337892 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.340169 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.340178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:35.344024 (+ 3846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:35.344059 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.345963 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.345973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:35.346414 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:35.347293 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:35.347319 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:35.350624 (+ 3305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:35.356368 (+ 5744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:35.356389 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:35.356415 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:35.357355 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:35.357880 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:35.357885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:35.357888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:35.358142 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:35.359016 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:35.359027 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:35.359646 (+ 619us) client_negotiation.cc:769] Sending connection context
0928 10:51:35.360092 (+ 446us) client_negotiation.cc:240] Negotiation successful
0928 10:51:35.361194 (+ 1102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":568,"threads_started":1}
I20260928 10:51:35.396556 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:35.375089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:49511 (local address 127.5.227.190:40249)
0928 10:51:35.375313 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:35.375319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:35.376217 (+ 898us) server_negotiation.cc:407] Connection header received
0928 10:51:35.376505 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:35.376510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:35.376590 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:35.376708 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:35.378653 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.380080 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.383230 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.384599 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.386111 (+ 1512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:35.388913 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:35.388923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:35.388930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:35.388978 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:35.392802 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:35.393799 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:35.393808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:35.393813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:35.394014 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:35.394599 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:35.394607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:35.394613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:35.395125 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:35.395319 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:35.395928 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 10:51:35.396171 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:51:35.396673 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:35.375189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40249 (local address 127.5.227.131:49511)
0928 10:51:35.376072 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:35.376098 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:35.376339 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:35.376731 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:35.376736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:35.377632 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:35.378360 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:35.378378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.380340 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.380348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:35.382916 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:35.382944 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.385202 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.385213 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:35.385607 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:35.386680 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:35.386704 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:35.388504 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:35.393013 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:35.393023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:35.393029 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:35.393639 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:35.394200 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:35.394208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:35.394217 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:35.394419 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:35.395297 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:35.395303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:35.395607 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 10:51:35.395810 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 10:51:35.396185 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":383,"threads_started":1}
I20260928 10:51:35.399665 13982 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:36011
I20260928 10:51:35.399816 14117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:36011 every 8 connection(s)
I20260928 10:51:35.401962 13982 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:35.404821 14118 heartbeater.cc:347] Connected to a master server at 127.5.227.190:40249
I20260928 10:51:35.405123 14118 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:35.406199 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 13982
I20260928 10:51:35.406201 14118 heartbeater.cc:511] Master 127.5.227.190:40249 requested a full tablet report, sending...
I20260928 10:51:35.406435 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:51:35.406986 6030 external_mini_cluster.cc:1428] Setting key 0581d72355a7adfa47f319ec5da2347a
I20260928 10:51:35.408475 13630 ts_manager.cc:194] Registered new tserver with Master: a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011)
I20260928 10:51:35.410070 13630 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:36997
I20260928 10:51:35.411099 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:51:35 dist-test-slave-8w8g krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592695, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:51:35 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:35 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:35.757895 13978 heartbeater.cc:503] Master 127.5.227.190:40249 was elected leader, sending a full tablet report...
I20260928 10:51:36.413218 14118 heartbeater.cc:503] Master 127.5.227.190:40249 was elected leader, sending a full tablet report...
W20260928 10:51:36.636055 14126 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.171s user 0.004s sys 0.013s
W20260928 10:51:36.637457 14005 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:35.458599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.5.227.190:40249)
0928 10:51:35.458892 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:35.458901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:35.459270 (+ 369us) server_negotiation.cc:407] Connection header received
0928 10:51:35.459510 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:35.459515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:35.459601 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:35.459722 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:35.461254 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.462036 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.463416 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:35.463955 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:35.464119 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.637033 (+1172914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48194: BlockingRecv error: recv got EOF from 127.0.0.1:48194 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 28 10:51:36 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592695, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:36.658260 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.642034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.5.227.190:40249)
0928 10:51:36.642294 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.642304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.642639 (+ 335us) server_negotiation.cc:407] Connection header received
0928 10:51:36.642875 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.642880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.642954 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.643064 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:36.644630 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.645436 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.647005 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.647471 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.647632 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.652272 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:36.652278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:36.652281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:36.652321 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:36.655100 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:36.655998 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:36.656002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:36.656005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:36.656157 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:36.656653 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:36.656657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:36.656660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:36.656982 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:36.657194 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.657758 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.657988 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 10:51:36.663384 13630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:51:36.670897 13630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:36.699411 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.685510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44351 (local address 127.0.0.1:40316)
0928 10:51:36.686804 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:36.686841 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:36.687014 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:36.691254 (+ 4240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:36.691261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:36.691396 (+ 135us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:36.691932 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.691943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.693856 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.693861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.696263 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.696273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.698245 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.698254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.698651 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.698684 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 10:51:36.698775 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 10:51:36.699021 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":189,"thread_start_us":826,"threads_started":1}
I20260928 10:51:36.699414 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.687507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.5.227.129:37363)
0928 10:51:36.689833 (+ 2326us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.689850 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.690632 (+ 782us) server_negotiation.cc:407] Connection header received
0928 10:51:36.690758 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.690766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.691038 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.691237 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:36.692373 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.693825 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.695955 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.697570 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.697729 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.697989 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.698957 (+ 968us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.699122 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":1068,"threads_started":1}
I20260928 10:51:36.699371 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.685616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.5.227.130:44351)
0928 10:51:36.690664 (+ 5048us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.690675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.690705 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:51:36.690817 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.690827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.691041 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.691245 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:36.692112 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.693625 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.696453 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.697949 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.698489 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.698745 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.698839 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.699021 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4885,"thread_start_us":4750,"threads_started":1}
I20260928 10:51:36.699390 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.687197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.5.227.131:36011)
0928 10:51:36.689291 (+ 2094us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.689305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.689346 (+ 41us) server_negotiation.cc:407] Connection header received
0928 10:51:36.689472 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.689485 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.689861 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.690778 (+ 917us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:36.690985 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.692942 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.696241 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.697743 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.697873 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.698040 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.698922 (+ 882us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.699102 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1863,"thread_start_us":918,"threads_started":1}
I20260928 10:51:36.700497 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.686561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:36011 (local address 127.0.0.1:58894)
0928 10:51:36.687646 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:36.687669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:36.687858 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:36.690191 (+ 2333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:36.690198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:36.690233 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:36.690747 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.690760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.693095 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.693100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.696005 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.696021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.698213 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.698223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.698496 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.698601 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 10:51:36.698771 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 10:51:36.699619 (+ 848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":688,"thread_start_us":800,"threads_started":1}
I20260928 10:51:36.700580 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.687276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37363 (local address 127.0.0.1:55750)
0928 10:51:36.690439 (+ 3163us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:36.690459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:36.690591 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:36.691442 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:36.691447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:36.691478 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:36.692167 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.692180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.693976 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.693982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.695703 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.695719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.698184 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.698192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.698600 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.698659 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 10:51:36.698764 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 10:51:36.699978 (+ 1214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2993,"mutex_wait_us":1127,"thread_start_us":2788,"threads_started":1}
I20260928 10:51:36.704373 13908 tablet_service.cc:1467] Processing CreateTablet for tablet 1807d30cb3f24e719f00c94ed59d7ea0 (DEFAULT_TABLE table=test-table [id=aabc5bd28ad04ba08b3093c6453c618e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:36.704664 14048 tablet_service.cc:1467] Processing CreateTablet for tablet 1807d30cb3f24e719f00c94ed59d7ea0 (DEFAULT_TABLE table=test-table [id=aabc5bd28ad04ba08b3093c6453c618e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:36.706166 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1807d30cb3f24e719f00c94ed59d7ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:36.706171 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1807d30cb3f24e719f00c94ed59d7ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:36.709720 13768 tablet_service.cc:1467] Processing CreateTablet for tablet 1807d30cb3f24e719f00c94ed59d7ea0 (DEFAULT_TABLE table=test-table [id=aabc5bd28ad04ba08b3093c6453c618e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:36.711608 13768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1807d30cb3f24e719f00c94ed59d7ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:36.720701 14143 tablet_bootstrap.cc:492] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Bootstrap starting.
I20260928 10:51:36.724638 14144 tablet_bootstrap.cc:492] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Bootstrap starting.
I20260928 10:51:36.727103 14143 tablet_bootstrap.cc:654] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:36.729000 14143 log.cc:824] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:36.729331 14144 tablet_bootstrap.cc:654] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:36.730955 14144 log.cc:824] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:36.734445 14145 tablet_bootstrap.cc:492] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Bootstrap starting.
I20260928 10:51:36.738494 14145 tablet_bootstrap.cc:654] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:36.739857 14143 tablet_bootstrap.cc:492] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: No bootstrap required, opened a new log
I20260928 10:51:36.739890 14144 tablet_bootstrap.cc:492] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: No bootstrap required, opened a new log
I20260928 10:51:36.740204 14143 ts_tablet_manager.cc:1397] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20260928 10:51:36.740350 14144 ts_tablet_manager.cc:1397] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260928 10:51:36.740442 14145 log.cc:824] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:36.754110 14143 raft_consensus.cc:348] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.754703 14143 raft_consensus.cc:374] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:36.754801 14143 raft_consensus.cc:729] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Initialized, Role: FOLLOWER
I20260928 10:51:36.754626 14144 raft_consensus.cc:348] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.755172 14145 tablet_bootstrap.cc:492] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: No bootstrap required, opened a new log
I20260928 10:51:36.755235 14144 raft_consensus.cc:374] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:36.755353 14144 raft_consensus.cc:729] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b45d7801c456da77fe8d6093af740, State: Initialized, Role: FOLLOWER
I20260928 10:51:36.755499 14145 ts_tablet_manager.cc:1397] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20260928 10:51:36.755555 14143 consensus_queue.cc:260] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.756033 14144 consensus_queue.cc:260] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.762039 14144 ts_tablet_manager.cc:1428] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Time spent starting tablet: real 0.022s user 0.013s sys 0.005s
I20260928 10:51:36.765223 14143 ts_tablet_manager.cc:1428] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Time spent starting tablet: real 0.025s user 0.014s sys 0.009s
I20260928 10:51:36.766266 14145 raft_consensus.cc:348] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.766889 14145 raft_consensus.cc:374] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:36.767051 14145 raft_consensus.cc:729] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae97524b417474c970c9daf1b3b0eda, State: Initialized, Role: FOLLOWER
I20260928 10:51:36.767815 14145 consensus_queue.cc:260] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.769951 14145 ts_tablet_manager.cc:1428] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260928 10:51:36.826196 14149 raft_consensus.cc:482] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:36.826675 14149 raft_consensus.cc:504] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.828519 14149 leader_election.cc:288] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351), aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363)
I20260928 10:51:36.829317 14150 raft_consensus.cc:482] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:36.829630 14150 raft_consensus.cc:504] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.836045 14150 leader_election.cc:288] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011)
I20260928 10:51:36.840116 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.828916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:55517 (local address 127.5.227.130:44351)
0928 10:51:36.829250 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.829259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.829874 (+ 615us) server_negotiation.cc:407] Connection header received
0928 10:51:36.830192 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.830201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.830307 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.830374 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:36.831307 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.832808 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.836481 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.838393 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.838569 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.838665 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.839455 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.839690 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:51:36.841542 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.828820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44351 (local address 127.5.227.131:55517)
0928 10:51:36.829751 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:36.829774 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:36.829999 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:36.830581 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:36.830585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:36.830618 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:36.831084 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.831098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.833033 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.833043 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.836149 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.836191 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.838883 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.838889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.839167 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.839223 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 10:51:36.840646 (+ 1423us) client_negotiation.cc:240] Negotiation successful
0928 10:51:36.840882 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":540,"threads_started":1}
I20260928 10:51:36.844017 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1807d30cb3f24e719f00c94ed59d7ea0" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b45d7801c456da77fe8d6093af740" is_pre_election: true
I20260928 10:51:36.844595 13928 raft_consensus.cc:2454] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 0.
I20260928 10:51:36.845615 14000 leader_election.cc:302] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f83f11adea4c869cfb4cc27c4e563f, af3b45d7801c456da77fe8d6093af740; no voters:
I20260928 10:51:36.846349 14149 raft_consensus.cc:2781] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:36.846557 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.830129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:41937 (local address 127.5.227.129:37363)
0928 10:51:36.830415 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.830423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.831134 (+ 711us) server_negotiation.cc:407] Connection header received
0928 10:51:36.831436 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.831441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.831524 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.831619 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:36.838833 (+ 7214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.840677 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.843896 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.845213 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.845363 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.845443 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.846144 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.846290 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 10:51:36.846601 14149 raft_consensus.cc:482] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:36.846720 14149 raft_consensus.cc:3037] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:36.852415 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.829878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37363 (local address 127.5.227.131:41937)
0928 10:51:36.831016 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:36.831035 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:36.831203 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:36.837661 (+ 6458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:36.837667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:36.837710 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:36.838541 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.838570 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.840983 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.840993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.843624 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.843644 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.845731 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.845737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.845927 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.845967 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:51:36.851797 (+ 5830us) client_negotiation.cc:240] Negotiation successful
0928 10:51:36.851977 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":800,"threads_started":1}
I20260928 10:51:36.852468 14149 raft_consensus.cc:504] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.854096 14149 leader_election.cc:288] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 election: Requested vote from peers af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351), aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363)
I20260928 10:51:36.853859 13793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1807d30cb3f24e719f00c94ed59d7ea0" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda" is_pre_election: true
I20260928 10:51:36.854542 13793 raft_consensus.cc:2454] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 0.
I20260928 10:51:36.854813 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.832574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:49563 (local address 127.5.227.131:36011)
0928 10:51:36.832819 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.832827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.843309 (+ 10482us) server_negotiation.cc:407] Connection header received
0928 10:51:36.843603 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.843612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.843713 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.843858 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:36.845025 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.846597 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.850481 (+ 3884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.852153 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.853734 (+ 1581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.853826 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.854385 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.854546 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":68}
I20260928 10:51:36.854770 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1807d30cb3f24e719f00c94ed59d7ea0" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda"
I20260928 10:51:36.855049 13792 raft_consensus.cc:3037] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:36.855638 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.832051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:36011 (local address 127.5.227.130:49563)
0928 10:51:36.843191 (+ 11140us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:36.843215 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:36.843426 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:36.844151 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:36.844158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:36.844211 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:36.844769 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.844787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.846907 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.846917 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.850200 (+ 3283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.850228 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.853983 (+ 3755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.853991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.854170 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.854210 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:51:36.855107 (+ 897us) client_negotiation.cc:240] Negotiation successful
0928 10:51:36.855262 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10956,"thread_start_us":10863,"threads_started":1}
I20260928 10:51:36.855680 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.832052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:46841 (local address 127.5.227.129:37363)
0928 10:51:36.846818 (+ 14766us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.846824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.846844 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:51:36.846902 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.846908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.846997 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.847087 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:51:36.848162 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.849237 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.852730 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.854367 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.854520 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.854588 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.855185 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.855354 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14642,"thread_start_us":19743,"threads_started":1}
I20260928 10:51:36.856432 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.831971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:37363 (local address 127.5.227.130:46841)
0928 10:51:36.842847 (+ 10876us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:36.842872 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:36.843037 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:36.847411 (+ 4374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:36.847416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:36.847463 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:51:36.847974 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.847989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.849653 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.849659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.852490 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:36.852508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.854780 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.854786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:36.854988 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.855026 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:51:36.855855 (+ 829us) client_negotiation.cc:240] Negotiation successful
0928 10:51:36.856008 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10697,"thread_start_us":10502,"threads_started":1}
I20260928 10:51:36.856909 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1807d30cb3f24e719f00c94ed59d7ea0" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" is_pre_election: true
I20260928 10:51:36.856370 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1807d30cb3f24e719f00c94ed59d7ea0" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b45d7801c456da77fe8d6093af740"
I20260928 10:51:36.857381 13928 raft_consensus.cc:3037] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:36.857506 14073 raft_consensus.cc:2379] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3b45d7801c456da77fe8d6093af740 in current term 1: Already voted for candidate a9f83f11adea4c869cfb4cc27c4e563f in this term.
I20260928 10:51:36.857964 13793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1807d30cb3f24e719f00c94ed59d7ea0" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda" is_pre_election: true
I20260928 10:51:36.858386 13792 raft_consensus.cc:2454] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 1.
I20260928 10:51:36.858824 13860 leader_election.cc:302] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3b45d7801c456da77fe8d6093af740; no voters: a9f83f11adea4c869cfb4cc27c4e563f, aae97524b417474c970c9daf1b3b0eda
I20260928 10:51:36.859015 14000 leader_election.cc:302] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f83f11adea4c869cfb4cc27c4e563f, aae97524b417474c970c9daf1b3b0eda; no voters:
I20260928 10:51:36.859515 14149 raft_consensus.cc:2781] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:36.860996 14149 raft_consensus.cc:686] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 LEADER]: Becoming Leader. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Running, Role: LEADER
I20260928 10:51:36.861676 13928 raft_consensus.cc:2454] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 1.
I20260928 10:51:36.861876 14150 raft_consensus.cc:2726] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 10:51:36.861799 14149 consensus_queue.cc:237] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.866928 13631 catalog_manager.cc:5495] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131). New cstate: current_term: 1 leader_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } health_report { overall_health: HEALTHY } } }
W20260928 10:51:36.880154 13839 tablet.cc:2369] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:36.901760 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:36.895277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.5.227.131:36011)
0928 10:51:36.895449 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:36.895454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:36.895513 (+ 59us) server_negotiation.cc:407] Connection header received
0928 10:51:36.895785 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:36.895798 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:36.895895 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:36.896038 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:36.896500 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.897397 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.899034 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:36.899516 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:36.899635 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:36.900335 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:36.901064 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:36.901258 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:36.901399 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 10:51:36.901481 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 10:51:36.905515 14119 tablet.cc:2369] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:36.912019 13792 raft_consensus.cc:1261] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Refusing update from remote peer a9f83f11adea4c869cfb4cc27c4e563f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:36.912082 13928 raft_consensus.cc:1261] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Refusing update from remote peer a9f83f11adea4c869cfb4cc27c4e563f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:36.913110 14157 consensus_queue.cc:1035] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:36.913702 14149 consensus_queue.cc:1035] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:36.926386 14162 mvcc.cc:205] Tried to move back new op lower bound from 7334267686539714560 to 7334267686351769600. Current Snapshot: MvccSnapshot[applied={T|T < 7334267686539714560}]
I20260928 10:51:36.926671 14160 mvcc.cc:205] Tried to move back new op lower bound from 7334267686539714560 to 7334267686351769600. Current Snapshot: MvccSnapshot[applied={T|T < 7334267686539714560}]
I20260928 10:51:36.944690 13631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:36.945107 13631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:36.949323 13631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Sending DeleteTablet for 3 replicas of tablet 1807d30cb3f24e719f00c94ed59d7ea0
I20260928 10:51:36.950888 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet 1807d30cb3f24e719f00c94ed59d7ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:36 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:58894
I20260928 10:51:36.950927 13768 tablet_service.cc:1514] Processing DeleteTablet for tablet 1807d30cb3f24e719f00c94ed59d7ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:36 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:55750
I20260928 10:51:36.951309 13908 tablet_service.cc:1514] Processing DeleteTablet for tablet 1807d30cb3f24e719f00c94ed59d7ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:36 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:40316
I20260928 10:51:36.952807 14171 tablet_replica.cc:330] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: stopping tablet replica
I20260928 10:51:36.953466 14171 raft_consensus.cc:2229] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:36.953792 13631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 10:51:36.954337 14171 raft_consensus.cc:2258] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 10:51:36.954624 13631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:36.955272 14170 tablet_replica.cc:330] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: stopping tablet replica
I20260928 10:51:36.955945 14170 raft_consensus.cc:2229] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:36.956555 14170 raft_consensus.cc:2258] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:36.957338 14171 ts_tablet_manager.cc:1905] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:36.958568 14170 ts_tablet_manager.cc:1905] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:36.966054 14171 ts_tablet_manager.cc:1918] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:36.966471 14171 log.cc:1196] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/1807d30cb3f24e719f00c94ed59d7ea0
I20260928 10:51:36.966575 14170 ts_tablet_manager.cc:1918] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:36.966795 14170 log.cc:1196] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/1807d30cb3f24e719f00c94ed59d7ea0
I20260928 10:51:36.967366 14170 ts_tablet_manager.cc:1939] T 1807d30cb3f24e719f00c94ed59d7ea0 P aae97524b417474c970c9daf1b3b0eda: Deleting consensus metadata
I20260928 10:51:36.967370 14171 ts_tablet_manager.cc:1939] T 1807d30cb3f24e719f00c94ed59d7ea0 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting consensus metadata
I20260928 10:51:36.967969 13768 tablet_service.cc:1467] Processing CreateTablet for tablet 91ec34a4c1e44fe0a1074be98418d91d (DEFAULT_TABLE table=test-table [id=99772dab318047a081a299e703593f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:36.968284 14048 tablet_service.cc:1467] Processing CreateTablet for tablet 91ec34a4c1e44fe0a1074be98418d91d (DEFAULT_TABLE table=test-table [id=99772dab318047a081a299e703593f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:36.968498 13768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ec34a4c1e44fe0a1074be98418d91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:36.968766 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ec34a4c1e44fe0a1074be98418d91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:36.972095 13617 catalog_manager.cc:4841] TS aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363): tablet 1807d30cb3f24e719f00c94ed59d7ea0 (table test-table [id=aabc5bd28ad04ba08b3093c6453c618e]) successfully deleted
I20260928 10:51:36.973230 13907 tablet_service.cc:1467] Processing CreateTablet for tablet 91ec34a4c1e44fe0a1074be98418d91d (DEFAULT_TABLE table=test-table [id=99772dab318047a081a299e703593f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:36.973932 13907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ec34a4c1e44fe0a1074be98418d91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:36.975286 14145 tablet_bootstrap.cc:492] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: Bootstrap starting.
I20260928 10:51:36.977694 14143 tablet_bootstrap.cc:492] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: Bootstrap starting.
I20260928 10:51:36.977759 14145 tablet_bootstrap.cc:654] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:36.979547 14143 tablet_bootstrap.cc:654] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:36.984203 13616 catalog_manager.cc:4841] TS a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011): tablet 1807d30cb3f24e719f00c94ed59d7ea0 (table test-table [id=aabc5bd28ad04ba08b3093c6453c618e]) successfully deleted
I20260928 10:51:36.985517 14144 tablet_bootstrap.cc:492] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Bootstrap starting.
I20260928 10:51:36.985991 14172 tablet_replica.cc:330] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: stopping tablet replica
I20260928 10:51:36.986524 14172 raft_consensus.cc:2229] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:36.987227 14172 raft_consensus.cc:2258] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:36.987519 14144 tablet_bootstrap.cc:654] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:36.992352 14145 tablet_bootstrap.cc:492] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: No bootstrap required, opened a new log
I20260928 10:51:36.992631 14145 ts_tablet_manager.cc:1397] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.001s
I20260928 10:51:36.993345 14143 tablet_bootstrap.cc:492] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: No bootstrap required, opened a new log
I20260928 10:51:36.993651 14143 ts_tablet_manager.cc:1397] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260928 10:51:36.993773 14145 raft_consensus.cc:348] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.994025 14145 raft_consensus.cc:374] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:36.994055 14172 ts_tablet_manager.cc:1905] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:36.994129 14145 raft_consensus.cc:729] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae97524b417474c970c9daf1b3b0eda, State: Initialized, Role: FOLLOWER
I20260928 10:51:36.994402 14145 consensus_queue.cc:260] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.995131 14145 ts_tablet_manager.cc:1428] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 10:51:36.996884 14143 raft_consensus.cc:348] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.997221 14143 raft_consensus.cc:374] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:36.997427 14143 raft_consensus.cc:729] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Initialized, Role: FOLLOWER
I20260928 10:51:36.997838 14143 consensus_queue.cc:260] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:36.999317 14143 ts_tablet_manager.cc:1428] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 10:51:37.001387 14144 tablet_bootstrap.cc:492] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: No bootstrap required, opened a new log
I20260928 10:51:37.001637 14144 ts_tablet_manager.cc:1397] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260928 10:51:37.002645 14144 raft_consensus.cc:348] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.003033 14144 raft_consensus.cc:374] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:37.003191 14144 raft_consensus.cc:729] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b45d7801c456da77fe8d6093af740, State: Initialized, Role: FOLLOWER
I20260928 10:51:37.003667 14144 consensus_queue.cc:260] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.005560 14144 ts_tablet_manager.cc:1428] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 10:51:37.006068 14172 ts_tablet_manager.cc:1918] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:37.006403 14172 log.cc:1196] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/1807d30cb3f24e719f00c94ed59d7ea0
I20260928 10:51:37.007162 14172 ts_tablet_manager.cc:1939] T 1807d30cb3f24e719f00c94ed59d7ea0 P af3b45d7801c456da77fe8d6093af740: Deleting consensus metadata
W20260928 10:51:37.007769 13979 tablet.cc:2369] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:51:37.009583 13617 catalog_manager.cc:4841] TS af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351): tablet 1807d30cb3f24e719f00c94ed59d7ea0 (table test-table [id=aabc5bd28ad04ba08b3093c6453c618e]) successfully deleted
I20260928 10:51:37.010748 14150 raft_consensus.cc:482] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:37.010933 14150 raft_consensus.cc:504] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.011636 14150 leader_election.cc:288] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011)
I20260928 10:51:37.012279 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ec34a4c1e44fe0a1074be98418d91d" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" is_pre_election: true
I20260928 10:51:37.012522 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ec34a4c1e44fe0a1074be98418d91d" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda" is_pre_election: true
I20260928 10:51:37.012569 14073 raft_consensus.cc:2454] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 0.
I20260928 10:51:37.012795 13792 raft_consensus.cc:2454] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 0.
I20260928 10:51:37.013170 13859 leader_election.cc:302] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f83f11adea4c869cfb4cc27c4e563f, af3b45d7801c456da77fe8d6093af740; no voters:
I20260928 10:51:37.013461 14150 raft_consensus.cc:2781] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:37.013592 14150 raft_consensus.cc:482] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:37.013686 14150 raft_consensus.cc:3037] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.017016 14150 raft_consensus.cc:504] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.018011 14150 leader_election.cc:288] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 election: Requested vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011)
I20260928 10:51:37.018910 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ec34a4c1e44fe0a1074be98418d91d" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda"
I20260928 10:51:37.019152 13792 raft_consensus.cc:3037] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.020040 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ec34a4c1e44fe0a1074be98418d91d" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f83f11adea4c869cfb4cc27c4e563f"
I20260928 10:51:37.020344 14073 raft_consensus.cc:3037] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.023069 13792 raft_consensus.cc:2454] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 1.
I20260928 10:51:37.023661 13860 leader_election.cc:302] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae97524b417474c970c9daf1b3b0eda, af3b45d7801c456da77fe8d6093af740; no voters:
I20260928 10:51:37.023862 14073 raft_consensus.cc:2454] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 1.
I20260928 10:51:37.024070 14150 raft_consensus.cc:2781] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:37.024573 14150 raft_consensus.cc:686] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 1 LEADER]: Becoming Leader. State: Replica: af3b45d7801c456da77fe8d6093af740, State: Running, Role: LEADER
I20260928 10:51:37.025107 14150 consensus_queue.cc:237] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.029702 13629 catalog_manager.cc:5495] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3b45d7801c456da77fe8d6093af740 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } health_report { overall_health: UNKNOWN } } }
Sep 28 10:51:37 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592693, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.190@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:37.088639 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.070810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:40249 (local address 127.0.0.1:48222)
0928 10:51:37.071582 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:37.071600 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:37.071762 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:37.072173 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:37.072178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:37.073153 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:37.073598 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:37.073622 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.075105 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.075111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:37.076852 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:37.076874 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.077794 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.077799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:37.078009 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.079003 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:37.079024 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:37.081768 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:37.085211 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:37.085229 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:37.085239 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:37.085729 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:37.086203 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:37.086209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:37.086213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:37.086435 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:37.087172 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:37.087182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:37.087432 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 10:51:37.087636 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 10:51:37.088018 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":426,"threads_started":1}
I20260928 10:51:37.088657 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.070672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.5.227.190:40249)
0928 10:51:37.070898 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:37.070904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:37.071701 (+ 797us) server_negotiation.cc:407] Connection header received
0928 10:51:37.071923 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:37.071928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:37.072004 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:37.072109 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:37.074046 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.074938 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.077036 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.077477 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.077723 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.082012 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:37.082017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:37.082021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:37.082053 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:37.085039 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:37.085877 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:37.085891 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:37.085895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:37.086017 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:37.086601 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:37.086606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:37.086608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:37.087006 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:37.087195 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:37.087882 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 10:51:37.088176 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":97}
I20260928 10:51:37.095034 13629 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:48222:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:51:37.096463 13629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:37.105036 13768 tablet_service.cc:1467] Processing CreateTablet for tablet 044ad5defcb44c07853cfcfb91d0b31c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15333a95ac98437298aa51c692f5ec3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:51:37.105535 14048 tablet_service.cc:1467] Processing CreateTablet for tablet 044ad5defcb44c07853cfcfb91d0b31c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15333a95ac98437298aa51c692f5ec3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:51:37.106019 13768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044ad5defcb44c07853cfcfb91d0b31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:37.106156 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044ad5defcb44c07853cfcfb91d0b31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:37.108567 13908 tablet_service.cc:1467] Processing CreateTablet for tablet 044ad5defcb44c07853cfcfb91d0b31c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15333a95ac98437298aa51c692f5ec3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:51:37.109419 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044ad5defcb44c07853cfcfb91d0b31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:37.116629 14143 tablet_bootstrap.cc:492] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: Bootstrap starting.
I20260928 10:51:37.118310 14144 tablet_bootstrap.cc:492] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: Bootstrap starting.
I20260928 10:51:37.118763 14143 tablet_bootstrap.cc:654] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:37.121006 14144 tablet_bootstrap.cc:654] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:37.121563 14145 tablet_bootstrap.cc:492] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: Bootstrap starting.
I20260928 10:51:37.123322 14145 tablet_bootstrap.cc:654] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:37.130653 14145 tablet_bootstrap.cc:492] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: No bootstrap required, opened a new log
I20260928 10:51:37.130908 14145 ts_tablet_manager.cc:1397] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260928 10:51:37.131487 14144 tablet_bootstrap.cc:492] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: No bootstrap required, opened a new log
I20260928 10:51:37.131711 14144 ts_tablet_manager.cc:1397] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260928 10:51:37.132046 14145 raft_consensus.cc:348] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.132359 14145 raft_consensus.cc:374] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:37.132470 14145 raft_consensus.cc:729] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae97524b417474c970c9daf1b3b0eda, State: Initialized, Role: FOLLOWER
I20260928 10:51:37.132664 14144 raft_consensus.cc:348] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.132959 14144 raft_consensus.cc:374] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:37.132874 14145 consensus_queue.cc:260] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.133051 14144 raft_consensus.cc:729] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b45d7801c456da77fe8d6093af740, State: Initialized, Role: FOLLOWER
I20260928 10:51:37.133335 14144 consensus_queue.cc:260] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.133525 14151 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.134004 14151 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:51:37.133862 14150 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.134199 14150 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:51:37.134763 14144 ts_tablet_manager.cc:1428] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 10:51:37.141433 14145 ts_tablet_manager.cc:1428] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20260928 10:51:37.144907 14143 tablet_bootstrap.cc:492] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: No bootstrap required, opened a new log
I20260928 10:51:37.145156 14143 ts_tablet_manager.cc:1397] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.012s
I20260928 10:51:37.146000 14143 raft_consensus.cc:348] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.146287 14143 raft_consensus.cc:374] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:37.146409 14143 raft_consensus.cc:729] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Initialized, Role: FOLLOWER
I20260928 10:51:37.146751 14143 consensus_queue.cc:260] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.147303 14157 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.147598 14143 ts_tablet_manager.cc:1428] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 10:51:37.147716 14157 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:51:37.251232 14150 raft_consensus.cc:482] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:37.251690 14150 raft_consensus.cc:504] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.252492 14150 leader_election.cc:288] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011)
I20260928 10:51:37.253082 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044ad5defcb44c07853cfcfb91d0b31c" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" is_pre_election: true
I20260928 10:51:37.253098 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044ad5defcb44c07853cfcfb91d0b31c" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda" is_pre_election: true
I20260928 10:51:37.253368 14073 raft_consensus.cc:2454] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 0.
I20260928 10:51:37.253407 13792 raft_consensus.cc:2454] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 0.
I20260928 10:51:37.253885 13860 leader_election.cc:302] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae97524b417474c970c9daf1b3b0eda, af3b45d7801c456da77fe8d6093af740; no voters:
I20260928 10:51:37.254187 14150 raft_consensus.cc:2781] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:37.254374 14150 raft_consensus.cc:482] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:37.254534 14150 raft_consensus.cc:3037] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.257131 14150 raft_consensus.cc:504] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.257891 14150 leader_election.cc:288] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 election: Requested vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011)
I20260928 10:51:37.258543 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044ad5defcb44c07853cfcfb91d0b31c" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f83f11adea4c869cfb4cc27c4e563f"
I20260928 10:51:37.258787 14073 raft_consensus.cc:3037] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.258649 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044ad5defcb44c07853cfcfb91d0b31c" candidate_uuid: "af3b45d7801c456da77fe8d6093af740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda"
I20260928 10:51:37.258992 13792 raft_consensus.cc:3037] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.261399 13792 raft_consensus.cc:2454] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 1.
I20260928 10:51:37.261950 13860 leader_election.cc:302] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae97524b417474c970c9daf1b3b0eda, af3b45d7801c456da77fe8d6093af740; no voters:
I20260928 10:51:37.262022 14073 raft_consensus.cc:2454] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3b45d7801c456da77fe8d6093af740 in term 1.
I20260928 10:51:37.262269 14150 raft_consensus.cc:2781] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:37.262565 14150 raft_consensus.cc:686] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [term 1 LEADER]: Becoming Leader. State: Replica: af3b45d7801c456da77fe8d6093af740, State: Running, Role: LEADER
I20260928 10:51:37.262913 14150 consensus_queue.cc:237] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.264856 14176 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: TxnStatusTablet state changed. Reason: New leader af3b45d7801c456da77fe8d6093af740. Latest consensus state: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.265286 14176 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:51:37.266471 14188 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:51:37.266767 14188 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:51:37.266896 13629 catalog_manager.cc:5495] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3b45d7801c456da77fe8d6093af740 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } health_report { overall_health: UNKNOWN } } }
I20260928 10:51:37.288959 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.281620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.5.227.130:44351)
0928 10:51:37.281830 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:37.281839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:37.282251 (+ 412us) server_negotiation.cc:407] Connection header received
0928 10:51:37.282590 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:37.282596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:37.282694 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:37.282863 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:37.283264 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.284264 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.285693 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.286393 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.286562 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.287169 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:37.287952 (+ 783us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:37.288177 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:37.288304 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 10:51:37.288400 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:51:37.288959 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.281800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44351 (local address 127.0.0.1:40328)
0928 10:51:37.282135 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:37.282154 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:37.282367 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:37.282887 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:37.282892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:37.282905 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:51:37.283109 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:37.283119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.284406 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.284414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:37.285527 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:37.285540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.286589 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.286598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:37.286808 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.286892 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:51:37.288151 (+ 1259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:51:37.288183 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:51:37.288279 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 10:51:37.288506 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 10:51:37.293807 13792 raft_consensus.cc:1261] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Refusing update from remote peer af3b45d7801c456da77fe8d6093af740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:37.294344 14073 raft_consensus.cc:1261] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Refusing update from remote peer af3b45d7801c456da77fe8d6093af740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:37.295229 14176 consensus_queue.cc:1035] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:37.296039 14150 consensus_queue.cc:1035] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:37.308898 14157 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: TxnStatusTablet state changed. Reason: New leader af3b45d7801c456da77fe8d6093af740. Latest consensus state: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.309549 14157 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:51:37.314901 14150 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.315419 14150 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:51:37.325459 14157 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.325866 14157 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P a9f83f11adea4c869cfb4cc27c4e563f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:51:37.333582 14151 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: TxnStatusTablet state changed. Reason: New leader af3b45d7801c456da77fe8d6093af740. Latest consensus state: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.333935 14151 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:51:37.336565 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.327253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.5.227.130:44351)
0928 10:51:37.328069 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:37.328080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:37.328106 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:51:37.328201 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:37.328211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:37.328507 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:37.328655 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:37.329178 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.330290 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.332449 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.334435 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.334596 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.335258 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:37.335413 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:37.336053 (+ 640us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:37.336111 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 10:51:37.336211 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616}
I20260928 10:51:37.337612 14176 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.337889 14176 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P af3b45d7801c456da77fe8d6093af740: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:51:37.352327 14151 tablet_replica.cc:439] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af3b45d7801c456da77fe8d6093af740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } } }
I20260928 10:51:37.352625 14151 tablet_replica.cc:442] T 044ad5defcb44c07853cfcfb91d0b31c P aae97524b417474c970c9daf1b3b0eda: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:51:37.383982 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.373044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:42981 (local address 127.5.227.130:44351)
0928 10:51:37.373232 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:37.373238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:37.376508 (+ 3270us) server_negotiation.cc:407] Connection header received
0928 10:51:37.376731 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:37.376739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:37.376842 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:37.376959 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:37.377541 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.379126 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.381367 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.382104 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.382282 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.382833 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:37.383052 (+ 219us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:37.383173 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:37.383341 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 10:51:37.383481 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":73}
I20260928 10:51:37.384083 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.372959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44351 (local address 127.5.227.130:42981)
0928 10:51:37.376343 (+ 3384us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:37.376369 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:37.376555 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:37.377006 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:37.377011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:37.377025 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:51:37.377308 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:37.377320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.379296 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.379302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:37.381168 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:37.381186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.382309 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.382316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:37.382515 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.382621 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:51:37.383187 (+ 566us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:51:37.383199 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:51:37.383294 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 10:51:37.383515 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3166,"mutex_wait_us":1,"thread_start_us":2977,"threads_started":1}
I20260928 10:51:37.398002 13792 raft_consensus.cc:1261] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Refusing update from remote peer af3b45d7801c456da77fe8d6093af740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:37.397989 14073 raft_consensus.cc:1261] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Refusing update from remote peer af3b45d7801c456da77fe8d6093af740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:37.398833 14176 consensus_queue.cc:1035] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:37.399169 14150 consensus_queue.cc:1035] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:37.423846 14161 mvcc.cc:205] Tried to move back new op lower bound from 7334267688535416832 to 7334267687019892736. Current Snapshot: MvccSnapshot[applied={T|T < 7334267688535416832}]
I20260928 10:51:37.532936 13629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:37.533200 13629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:37.537213 13629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Sending DeleteTablet for 3 replicas of tablet 91ec34a4c1e44fe0a1074be98418d91d
I20260928 10:51:37.538784 13768 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ec34a4c1e44fe0a1074be98418d91d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:37 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:55750
I20260928 10:51:37.538805 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ec34a4c1e44fe0a1074be98418d91d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:37 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:58894
I20260928 10:51:37.549247 14203 tablet_replica.cc:330] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: stopping tablet replica
I20260928 10:51:37.549708 13908 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ec34a4c1e44fe0a1074be98418d91d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:37 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:40316
I20260928 10:51:37.550341 14203 raft_consensus.cc:2229] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:37.551096 14203 raft_consensus.cc:2258] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:37.553372 14203 ts_tablet_manager.cc:1905] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:37.554980 14202 tablet_replica.cc:330] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: stopping tablet replica
I20260928 10:51:37.560855 14204 tablet_replica.cc:330] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: stopping tablet replica
I20260928 10:51:37.561115 14204 raft_consensus.cc:2229] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:37.561247 14202 raft_consensus.cc:2229] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:37.561648 14202 raft_consensus.cc:2258] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:37.561787 14204 raft_consensus.cc:2258] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:37.563251 14202 ts_tablet_manager.cc:1905] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:37.566421 14204 ts_tablet_manager.cc:1905] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:37.578295 14204 ts_tablet_manager.cc:1918] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:51:37.578483 14204 log.cc:1196] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/91ec34a4c1e44fe0a1074be98418d91d
I20260928 10:51:37.579119 14204 ts_tablet_manager.cc:1939] T 91ec34a4c1e44fe0a1074be98418d91d P af3b45d7801c456da77fe8d6093af740: Deleting consensus metadata
I20260928 10:51:37.580886 13617 catalog_manager.cc:4841] TS af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351): tablet 91ec34a4c1e44fe0a1074be98418d91d (table test-table [id=99772dab318047a081a299e703593f08]) successfully deleted
I20260928 10:51:37.583153 14202 ts_tablet_manager.cc:1918] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:51:37.583375 14202 log.cc:1196] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/91ec34a4c1e44fe0a1074be98418d91d
I20260928 10:51:37.583941 14202 ts_tablet_manager.cc:1939] T 91ec34a4c1e44fe0a1074be98418d91d P a9f83f11adea4c869cfb4cc27c4e563f: Deleting consensus metadata
I20260928 10:51:37.585203 14203 ts_tablet_manager.cc:1918] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:51:37.585217 13616 catalog_manager.cc:4841] TS a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011): tablet 91ec34a4c1e44fe0a1074be98418d91d (table test-table [id=99772dab318047a081a299e703593f08]) successfully deleted
I20260928 10:51:37.585656 14203 log.cc:1196] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/91ec34a4c1e44fe0a1074be98418d91d
I20260928 10:51:37.586222 14203 ts_tablet_manager.cc:1939] T 91ec34a4c1e44fe0a1074be98418d91d P aae97524b417474c970c9daf1b3b0eda: Deleting consensus metadata
I20260928 10:51:37.587626 13617 catalog_manager.cc:4841] TS aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363): tablet 91ec34a4c1e44fe0a1074be98418d91d (table test-table [id=99772dab318047a081a299e703593f08]) successfully deleted
Sep 28 10:51:37 dist-test-slave-8w8g krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:51:37 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:37 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:51:37.638191 14211 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:37.621854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.5.227.190:40249)
0928 10:51:37.623442 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:37.623450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:37.623920 (+ 470us) server_negotiation.cc:407] Connection header received
0928 10:51:37.624100 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:37.624106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:37.624212 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:37.624428 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:37.626377 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.627570 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.629325 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.630135 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.630317 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.637857 (+ 7540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48238: BlockingRecv error: recv got EOF from 127.0.0.1:48238 (error 108)
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":1344,"threads_started":1}
Sep 28 10:51:37 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM
I20260928 10:51:37.661448 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.642836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.5.227.190:40249)
0928 10:51:37.643017 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:37.643022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:37.643161 (+ 139us) server_negotiation.cc:407] Connection header received
0928 10:51:37.643418 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:37.643422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:37.643490 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:37.643770 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:37.645322 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.646200 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.647764 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.648444 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.648625 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.653408 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:37.653418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:37.653425 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:37.653481 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:37.657814 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:37.658965 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:37.658972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:37.658976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:37.659170 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:37.659647 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:37.659653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:37.659656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:37.660091 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:37.660363 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:37.660811 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 10:51:37.661063 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:51:37.666121 13629 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:51:37.666973 13629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:37.675894 13768 tablet_service.cc:1467] Processing CreateTablet for tablet bed583ac8d374692a9f042bd3c9f7fe1 (DEFAULT_TABLE table=test-table [id=f5c78d5d32294384a2a28339b82aa663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:37.675987 14048 tablet_service.cc:1467] Processing CreateTablet for tablet bed583ac8d374692a9f042bd3c9f7fe1 (DEFAULT_TABLE table=test-table [id=f5c78d5d32294384a2a28339b82aa663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:37.676425 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed583ac8d374692a9f042bd3c9f7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:37.676424 13768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed583ac8d374692a9f042bd3c9f7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:37.680176 13908 tablet_service.cc:1467] Processing CreateTablet for tablet bed583ac8d374692a9f042bd3c9f7fe1 (DEFAULT_TABLE table=test-table [id=f5c78d5d32294384a2a28339b82aa663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:37.680701 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed583ac8d374692a9f042bd3c9f7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:37.684191 14218 tablet_bootstrap.cc:492] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: Bootstrap starting.
I20260928 10:51:37.686188 14218 tablet_bootstrap.cc:654] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:37.693454 14221 tablet_bootstrap.cc:492] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: Bootstrap starting.
I20260928 10:51:37.695755 14218 tablet_bootstrap.cc:492] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: No bootstrap required, opened a new log
I20260928 10:51:37.696066 14220 tablet_bootstrap.cc:492] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: Bootstrap starting.
I20260928 10:51:37.696097 14218 ts_tablet_manager.cc:1397] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260928 10:51:37.697129 14218 raft_consensus.cc:348] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.697479 14218 raft_consensus.cc:374] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:37.697589 14218 raft_consensus.cc:729] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Initialized, Role: FOLLOWER
I20260928 10:51:37.697898 14218 consensus_queue.cc:260] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.698606 14220 tablet_bootstrap.cc:654] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:37.699432 14221 tablet_bootstrap.cc:654] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:37.703019 14218 ts_tablet_manager.cc:1428] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260928 10:51:37.710547 14220 tablet_bootstrap.cc:492] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: No bootstrap required, opened a new log
I20260928 10:51:37.710767 14220 ts_tablet_manager.cc:1397] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260928 10:51:37.711766 14220 raft_consensus.cc:348] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.712061 14220 raft_consensus.cc:374] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:37.712183 14220 raft_consensus.cc:729] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b45d7801c456da77fe8d6093af740, State: Initialized, Role: FOLLOWER
I20260928 10:51:37.712508 14220 consensus_queue.cc:260] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.713205 14220 ts_tablet_manager.cc:1428] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 10:51:37.721223 14221 tablet_bootstrap.cc:492] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: No bootstrap required, opened a new log
I20260928 10:51:37.721529 14221 ts_tablet_manager.cc:1397] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.011s
I20260928 10:51:37.722235 14221 raft_consensus.cc:348] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.722499 14221 raft_consensus.cc:374] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:37.722627 14221 raft_consensus.cc:729] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae97524b417474c970c9daf1b3b0eda, State: Initialized, Role: FOLLOWER
I20260928 10:51:37.722893 14221 consensus_queue.cc:260] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.723794 14221 ts_tablet_manager.cc:1428] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 10:51:37.914089 14157 raft_consensus.cc:482] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:37.914388 14157 raft_consensus.cc:504] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.915119 14157 leader_election.cc:288] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351)
I20260928 10:51:37.915930 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed583ac8d374692a9f042bd3c9f7fe1" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b45d7801c456da77fe8d6093af740" is_pre_election: true
I20260928 10:51:37.915928 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed583ac8d374692a9f042bd3c9f7fe1" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda" is_pre_election: true
I20260928 10:51:37.916246 13928 raft_consensus.cc:2454] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 0.
I20260928 10:51:37.916213 13792 raft_consensus.cc:2454] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 0.
I20260928 10:51:37.916903 14000 leader_election.cc:302] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f83f11adea4c869cfb4cc27c4e563f, aae97524b417474c970c9daf1b3b0eda; no voters:
I20260928 10:51:37.917337 14157 raft_consensus.cc:2781] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:37.917469 14157 raft_consensus.cc:482] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:37.917635 14157 raft_consensus.cc:3037] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.920079 14157 raft_consensus.cc:504] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.920727 14157 leader_election.cc:288] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 election: Requested vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351)
I20260928 10:51:37.921454 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed583ac8d374692a9f042bd3c9f7fe1" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b45d7801c456da77fe8d6093af740"
I20260928 10:51:37.921522 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed583ac8d374692a9f042bd3c9f7fe1" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda"
I20260928 10:51:37.921725 13928 raft_consensus.cc:3037] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.921893 13792 raft_consensus.cc:3037] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:37.924234 13928 raft_consensus.cc:2454] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 1.
I20260928 10:51:37.924808 14000 leader_election.cc:302] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f83f11adea4c869cfb4cc27c4e563f, af3b45d7801c456da77fe8d6093af740; no voters:
I20260928 10:51:37.925136 14157 raft_consensus.cc:2781] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:37.925546 13792 raft_consensus.cc:2454] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 1.
I20260928 10:51:37.925697 14157 raft_consensus.cc:686] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 LEADER]: Becoming Leader. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Running, Role: LEADER
I20260928 10:51:37.926186 14157 consensus_queue.cc:237] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:37.938270 13630 catalog_manager.cc:5495] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131). New cstate: current_term: 1 leader_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } health_report { overall_health: HEALTHY } } }
I20260928 10:51:38.000547 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:37.992323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.5.227.131:36011)
0928 10:51:37.992987 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:37.992995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:37.993018 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:51:37.993097 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:37.993102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:37.993195 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:37.993319 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:37.993960 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.995320 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.997572 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:37.998444 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:37.998609 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:37.999287 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:37.999482 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:37.999639 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:38.000005 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 10:51:38.000108 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":318,"threads_started":1}
I20260928 10:51:38.004334 13792 raft_consensus.cc:1261] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Refusing update from remote peer a9f83f11adea4c869cfb4cc27c4e563f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:38.004280 13928 raft_consensus.cc:1261] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Refusing update from remote peer a9f83f11adea4c869cfb4cc27c4e563f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:38.005023 14224 consensus_queue.cc:1035] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:38.005542 14157 consensus_queue.cc:1035] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:38.019936 13631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:38.020251 13631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:38.026415 13631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Sending DeleteTablet for 3 replicas of tablet bed583ac8d374692a9f042bd3c9f7fe1
I20260928 10:51:38.027524 13768 tablet_service.cc:1514] Processing DeleteTablet for tablet bed583ac8d374692a9f042bd3c9f7fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:38 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:55750
I20260928 10:51:38.028039 14203 tablet_replica.cc:330] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: stopping tablet replica
I20260928 10:51:38.028512 14203 raft_consensus.cc:2229] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:38.028860 14203 raft_consensus.cc:2258] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:38.028925 13908 tablet_service.cc:1514] Processing DeleteTablet for tablet bed583ac8d374692a9f042bd3c9f7fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:38 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:40316
I20260928 10:51:38.029630 14204 tablet_replica.cc:330] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: stopping tablet replica
I20260928 10:51:38.029942 14204 raft_consensus.cc:2229] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:38.030229 14204 raft_consensus.cc:2258] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:38.031502 14203 ts_tablet_manager.cc:1905] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:38.031571 14204 ts_tablet_manager.cc:1905] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:38.031981 13630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:51:38.033001 13630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:51:38.040875 14204 ts_tablet_manager.cc:1918] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:38.040875 14203 ts_tablet_manager.cc:1918] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:38.041075 14203 log.cc:1196] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/bed583ac8d374692a9f042bd3c9f7fe1
I20260928 10:51:38.041080 14204 log.cc:1196] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/bed583ac8d374692a9f042bd3c9f7fe1
I20260928 10:51:38.041575 14204 ts_tablet_manager.cc:1939] T bed583ac8d374692a9f042bd3c9f7fe1 P af3b45d7801c456da77fe8d6093af740: Deleting consensus metadata
I20260928 10:51:38.041604 14203 ts_tablet_manager.cc:1939] T bed583ac8d374692a9f042bd3c9f7fe1 P aae97524b417474c970c9daf1b3b0eda: Deleting consensus metadata
I20260928 10:51:38.042922 13617 catalog_manager.cc:4841] TS af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351): tablet bed583ac8d374692a9f042bd3c9f7fe1 (table test-table [id=f5c78d5d32294384a2a28339b82aa663]) successfully deleted
I20260928 10:51:38.043218 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet bed583ac8d374692a9f042bd3c9f7fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:38 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:58894
I20260928 10:51:38.043531 14202 tablet_replica.cc:330] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: stopping tablet replica
I20260928 10:51:38.043818 14202 raft_consensus.cc:2229] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:38.044011 13768 tablet_service.cc:1467] Processing CreateTablet for tablet a2641b18e07e425eac64b10d4454be29 (DEFAULT_TABLE table=test-table [id=7802022242b145fb90666e4a479871a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:38.044570 13768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2641b18e07e425eac64b10d4454be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:38.045578 14202 raft_consensus.cc:2258] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:38.048278 13617 catalog_manager.cc:4841] TS aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363): tablet bed583ac8d374692a9f042bd3c9f7fe1 (table test-table [id=f5c78d5d32294384a2a28339b82aa663]) successfully deleted
I20260928 10:51:38.048816 14048 tablet_service.cc:1467] Processing CreateTablet for tablet a2641b18e07e425eac64b10d4454be29 (DEFAULT_TABLE table=test-table [id=7802022242b145fb90666e4a479871a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:38.049119 13908 tablet_service.cc:1467] Processing CreateTablet for tablet a2641b18e07e425eac64b10d4454be29 (DEFAULT_TABLE table=test-table [id=7802022242b145fb90666e4a479871a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:51:38.049423 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2641b18e07e425eac64b10d4454be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:38.049678 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2641b18e07e425eac64b10d4454be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:38.051887 14221 tablet_bootstrap.cc:492] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: Bootstrap starting.
I20260928 10:51:38.053614 14221 tablet_bootstrap.cc:654] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:38.057649 14202 ts_tablet_manager.cc:1905] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:38.064523 14220 tablet_bootstrap.cc:492] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: Bootstrap starting.
I20260928 10:51:38.066807 14220 tablet_bootstrap.cc:654] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:38.069520 14221 tablet_bootstrap.cc:492] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: No bootstrap required, opened a new log
I20260928 10:51:38.069720 14221 ts_tablet_manager.cc:1397] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20260928 10:51:38.071688 14202 ts_tablet_manager.cc:1918] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:51:38.070555 14221 raft_consensus.cc:348] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.072573 14221 raft_consensus.cc:374] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:38.072680 14221 raft_consensus.cc:729] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae97524b417474c970c9daf1b3b0eda, State: Initialized, Role: FOLLOWER
I20260928 10:51:38.073004 14221 consensus_queue.cc:260] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.073365 14218 tablet_bootstrap.cc:492] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: Bootstrap starting.
I20260928 10:51:38.073616 14202 log.cc:1196] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/bed583ac8d374692a9f042bd3c9f7fe1
I20260928 10:51:38.074406 14202 ts_tablet_manager.cc:1939] T bed583ac8d374692a9f042bd3c9f7fe1 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting consensus metadata
I20260928 10:51:38.075722 14218 tablet_bootstrap.cc:654] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:38.078562 14221 ts_tablet_manager.cc:1428] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20260928 10:51:38.079172 13616 catalog_manager.cc:4841] TS a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011): tablet bed583ac8d374692a9f042bd3c9f7fe1 (table test-table [id=f5c78d5d32294384a2a28339b82aa663]) successfully deleted
I20260928 10:51:38.080930 14220 tablet_bootstrap.cc:492] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: No bootstrap required, opened a new log
I20260928 10:51:38.081142 14220 ts_tablet_manager.cc:1397] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260928 10:51:38.082139 14220 raft_consensus.cc:348] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.082459 14220 raft_consensus.cc:374] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:38.082552 14220 raft_consensus.cc:729] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b45d7801c456da77fe8d6093af740, State: Initialized, Role: FOLLOWER
I20260928 10:51:38.082880 14220 consensus_queue.cc:260] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.083431 14218 tablet_bootstrap.cc:492] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: No bootstrap required, opened a new log
I20260928 10:51:38.083613 14218 ts_tablet_manager.cc:1397] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260928 10:51:38.083684 14220 ts_tablet_manager.cc:1428] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 10:51:38.084479 14218 raft_consensus.cc:348] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.084762 14218 raft_consensus.cc:374] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:38.084859 14218 raft_consensus.cc:729] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Initialized, Role: FOLLOWER
I20260928 10:51:38.085134 14218 consensus_queue.cc:260] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.085821 14218 ts_tablet_manager.cc:1428] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 10:51:38.225709 14224 raft_consensus.cc:482] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:51:38.226027 14224 raft_consensus.cc:504] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.227145 14224 leader_election.cc:288] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351)
I20260928 10:51:38.228186 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2641b18e07e425eac64b10d4454be29" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b45d7801c456da77fe8d6093af740" is_pre_election: true
I20260928 10:51:38.228158 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2641b18e07e425eac64b10d4454be29" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda" is_pre_election: true
I20260928 10:51:38.228451 13792 raft_consensus.cc:2454] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 0.
I20260928 10:51:38.228513 13928 raft_consensus.cc:2454] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 0.
I20260928 10:51:38.229264 14000 leader_election.cc:302] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f83f11adea4c869cfb4cc27c4e563f, aae97524b417474c970c9daf1b3b0eda; no voters:
I20260928 10:51:38.229724 14224 raft_consensus.cc:2781] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:51:38.229851 14224 raft_consensus.cc:482] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:51:38.229978 14224 raft_consensus.cc:3037] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:38.233016 14224 raft_consensus.cc:504] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.234421 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2641b18e07e425eac64b10d4454be29" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae97524b417474c970c9daf1b3b0eda"
I20260928 10:51:38.234710 13792 raft_consensus.cc:3037] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:38.236074 13928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2641b18e07e425eac64b10d4454be29" candidate_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b45d7801c456da77fe8d6093af740"
I20260928 10:51:38.236477 13928 raft_consensus.cc:3037] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:38.238241 13792 raft_consensus.cc:2454] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 1.
I20260928 10:51:38.238780 14000 leader_election.cc:302] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f83f11adea4c869cfb4cc27c4e563f, aae97524b417474c970c9daf1b3b0eda; no voters:
I20260928 10:51:38.239169 14157 raft_consensus.cc:2781] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:38.239356 14157 raft_consensus.cc:686] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 LEADER]: Becoming Leader. State: Replica: a9f83f11adea4c869cfb4cc27c4e563f, State: Running, Role: LEADER
I20260928 10:51:38.239638 14157 consensus_queue.cc:237] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } }
I20260928 10:51:38.241127 13928 raft_consensus.cc:2454] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f83f11adea4c869cfb4cc27c4e563f in term 1.
I20260928 10:51:38.241865 14224 leader_election.cc:288] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [CANDIDATE]: Term 1 election: Requested vote from peers aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363), af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351)
I20260928 10:51:38.245105 13630 catalog_manager.cc:5495] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131). New cstate: current_term: 1 leader_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9f83f11adea4c869cfb4cc27c4e563f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 36011 } health_report { overall_health: HEALTHY } } }
I20260928 10:51:38.331645 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:38.310809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:38391 (local address 127.5.227.130:44351)
0928 10:51:38.323338 (+ 12529us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:38.323349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:38.324156 (+ 807us) server_negotiation.cc:407] Connection header received
0928 10:51:38.324482 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:38.324485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:38.324563 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:38.324677 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:38.325469 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.326588 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.329081 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.329600 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.329806 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:38.330355 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:38.330475 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:38.330584 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:38.330751 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 10:51:38.330842 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12104,"thread_start_us":12393,"threads_started":1}
I20260928 10:51:38.332075 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:38.310313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44351 (local address 127.5.227.131:38391)
0928 10:51:38.324005 (+ 13692us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:38.324026 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:38.324255 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:38.324756 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:38.324763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:38.324777 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:51:38.325266 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:38.325278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.326752 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.326758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:38.328894 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:38.328913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.329880 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.329894 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:38.330039 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:38.330110 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:51:38.330585 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:51:38.330600 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 10:51:38.330709 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 10:51:38.330895 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13493,"thread_start_us":13281,"threads_started":1}
I20260928 10:51:38.346223 13928 raft_consensus.cc:1261] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Refusing update from remote peer a9f83f11adea4c869cfb4cc27c4e563f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:38.346946 14224 consensus_queue.cc:1035] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3b45d7801c456da77fe8d6093af740" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:38.349751 13792 raft_consensus.cc:1261] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Refusing update from remote peer a9f83f11adea4c869cfb4cc27c4e563f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:51:38.350509 14224 consensus_queue.cc:1035] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae97524b417474c970c9daf1b3b0eda" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:51:38.432677 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:38.415040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:37373 (local address 127.5.227.131:36011)
0928 10:51:38.415321 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:38.415331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:38.424058 (+ 8727us) server_negotiation.cc:407] Connection header received
0928 10:51:38.424283 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:38.424292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:38.424414 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:38.424537 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:51:38.425358 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.426587 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.429634 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.430946 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.431158 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:38.431721 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:51:38.431920 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:51:38.431992 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:38.432214 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 10:51:38.432370 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:51:38.433745 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:38.414495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:36011 (local address 127.5.227.130:37373)
0928 10:51:38.423900 (+ 9405us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:38.423924 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:38.424093 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:38.424586 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:38.424591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:38.424606 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:51:38.425098 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:38.425115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.426865 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.426871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:38.429388 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:38.429410 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.431223 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.431232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:38.431440 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:38.431504 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:51:38.432081 (+ 577us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:51:38.432091 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 10:51:38.433094 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0928 10:51:38.433274 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9158,"thread_start_us":8970,"threads_started":1}
I20260928 10:51:38.505604 13630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:38.506011 13630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:51:38.510504 13630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc62d2466097448abf283907400e0b6b: Sending DeleteTablet for 3 replicas of tablet a2641b18e07e425eac64b10d4454be29
I20260928 10:51:38.511973 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet a2641b18e07e425eac64b10d4454be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:38 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:58894
I20260928 10:51:38.512521 14202 tablet_replica.cc:330] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: stopping tablet replica
I20260928 10:51:38.512791 14202 raft_consensus.cc:2229] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:51:38.513307 14202 raft_consensus.cc:2258] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:38.514345 13768 tablet_service.cc:1514] Processing DeleteTablet for tablet a2641b18e07e425eac64b10d4454be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:38 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:55750
I20260928 10:51:38.514705 14203 tablet_replica.cc:330] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: stopping tablet replica
I20260928 10:51:38.514886 14203 raft_consensus.cc:2229] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:38.515120 14203 raft_consensus.cc:2258] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:38.515935 14202 ts_tablet_manager.cc:1905] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:38.516503 14203 ts_tablet_manager.cc:1905] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:38.520066 13908 tablet_service.cc:1514] Processing DeleteTablet for tablet a2641b18e07e425eac64b10d4454be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:51:38 UTC) from {username='oryx', principal='oryx/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:40316
I20260928 10:51:38.520504 14204 tablet_replica.cc:330] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: stopping tablet replica
I20260928 10:51:38.520717 14204 raft_consensus.cc:2229] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:51:38.520995 14204 raft_consensus.cc:2258] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:51:38.522816 14204 ts_tablet_manager.cc:1905] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:51:38.523789 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 13702
I20260928 10:51:38.532675 14202 ts_tablet_manager.cc:1918] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:51:38.533026 14202 log.cc:1196] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/a2641b18e07e425eac64b10d4454be29
I20260928 10:51:38.533690 14202 ts_tablet_manager.cc:1939] T a2641b18e07e425eac64b10d4454be29 P a9f83f11adea4c869cfb4cc27c4e563f: Deleting consensus metadata
I20260928 10:51:38.535686 13616 catalog_manager.cc:4841] TS a9f83f11adea4c869cfb4cc27c4e563f (127.5.227.131:36011): tablet a2641b18e07e425eac64b10d4454be29 (table test-table [id=7802022242b145fb90666e4a479871a6]) successfully deleted
I20260928 10:51:38.540849 14203 ts_tablet_manager.cc:1918] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:51:38.541213 14203 log.cc:1196] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/a2641b18e07e425eac64b10d4454be29
I20260928 10:51:38.542357 14203 ts_tablet_manager.cc:1939] T a2641b18e07e425eac64b10d4454be29 P aae97524b417474c970c9daf1b3b0eda: Deleting consensus metadata
I20260928 10:51:38.544822 14204 ts_tablet_manager.cc:1918] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:51:38.545022 14204 log.cc:1196] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/a2641b18e07e425eac64b10d4454be29
I20260928 10:51:38.545543 14204 ts_tablet_manager.cc:1939] T a2641b18e07e425eac64b10d4454be29 P af3b45d7801c456da77fe8d6093af740: Deleting consensus metadata
I20260928 10:51:38.546885 13617 catalog_manager.cc:4841] TS af3b45d7801c456da77fe8d6093af740 (127.5.227.130:44351): tablet a2641b18e07e425eac64b10d4454be29 (table test-table [id=7802022242b145fb90666e4a479871a6]) successfully deleted
I20260928 10:51:38.547376 13617 catalog_manager.cc:4841] TS aae97524b417474c970c9daf1b3b0eda (127.5.227.129:37363): tablet a2641b18e07e425eac64b10d4454be29 (table test-table [id=7802022242b145fb90666e4a479871a6]) successfully deleted
Sep 28 10:51:38 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.129@KRBTEST.COM
I20260928 10:51:38.580016 14249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:38.525434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.5.227.129:37363)
0928 10:51:38.544043 (+ 18609us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:38.544052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:38.544078 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:51:38.544151 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:38.544158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:38.545278 (+ 1120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:38.545435 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:38.549850 (+ 4415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.551110 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.553755 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.554633 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.554984 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:38.568389 (+ 13405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:38.568426 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:38.568455 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:38.568522 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:38.574130 (+ 5608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:38.575655 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:38.575667 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:38.575670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:38.575909 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:38.576722 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:38.576729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:38.576733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:38.577334 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:38.577789 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:38.579218 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0928 10:51:38.579535 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18366,"thread_start_us":21685,"threads_started":1}
I20260928 10:51:38.580716 13833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:38.726094 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 13702
I20260928 10:51:38.745754 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 13842
Sep 28 10:51:38 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.130@KRBTEST.COM
I20260928 10:51:38.766387 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:38.746671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.5.227.130:44351)
0928 10:51:38.746878 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:38.746882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:38.746900 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:51:38.747169 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:38.747175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:38.747267 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:38.747368 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:38.749106 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.749961 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.752311 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.752803 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.752925 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:38.758776 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:38.758794 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:38.758812 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:38.758856 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:38.762394 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:38.763839 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:38.763853 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:38.763860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:38.764097 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:38.764682 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:38.764686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:38.764688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:38.765122 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:38.765382 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:38.765782 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 10:51:38.765989 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 10:51:38.767153 13973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:38.922119 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 13842
I20260928 10:51:38.952064 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 13982
Sep 28 10:51:38 dist-test-slave-8w8g krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.227.131@KRBTEST.COM
I20260928 10:51:38.970840 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:38.952931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.5.227.131:36011)
0928 10:51:38.953444 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:38.953450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:38.953472 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:51:38.953533 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:38.953536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:38.953644 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:38.953773 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:38.955225 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.956043 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.957415 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:38.957862 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:38.958015 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:38.963266 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:38.963284 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:38.963302 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:38.963347 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:38.966572 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:38.967513 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:38.967523 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:38.967525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:38.968050 (+ 525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:38.968817 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:38.968823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:38.968828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:38.969386 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:38.969741 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:38.970188 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 10:51:38.970450 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260928 10:51:38.971663 14113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:39.106856 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 13982
I20260928 10:51:39.133215 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 13599
I20260928 10:51:39.135046 13671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:39.264979 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 13599
2026-09-28T10:51:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (8051 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:51:39 dist-test-slave-8w8g krb5kdc[14257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:51:39 dist-test-slave-8w8g krb5kdc[14257](info): set up 2 sockets
Sep 28 10:51:39 dist-test-slave-8w8g krb5kdc[14257](info): commencing operation
krb5kdc: starting...
W20260928 10:51:40.466565 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:51:40 dist-test-slave-8w8g krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:51:40.742267 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40683
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:40683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:41.116973 14270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:41.117368 14270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:41.117542 14270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:41.126135 14270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:41.126271 14270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:41.126364 14270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:41.126461 14270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:41.126560 14270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:41.137327 14270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:40683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:41.138979 14270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:41.141120 14270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:41.157056 14278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:41.157171 14275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:41.157296 14276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:41.157824 14270 server_base.cc:1034] running on GCE node
I20260928 10:51:41.159070 14270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:41.161350 14270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:41.162686 14270 hybrid_clock.cc:648] HybridClock initialized: now 1790592701162606 us; error 67 us; skew 500 ppm
I20260928 10:51:41.163105 14270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T10:51:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1975 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:51:41 dist-test-slave-8w8g krb5kdc[14284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:51:41 dist-test-slave-8w8g krb5kdc[14284](info): set up 2 sockets
Sep 28 10:51:41 dist-test-slave-8w8g krb5kdc[14284](info): commencing operation
krb5kdc: starting...
W20260928 10:51:42.429543 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:51:42 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:51:42.689078 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:33507
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:33507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:43.043684 14297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:43.044140 14297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:43.044307 14297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:43.051815 14297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:43.051942 14297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:43.052023 14297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:43.052112 14297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:43.052152 14297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:43.062412 14297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:33507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:33507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:43.063843 14297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:43.065157 14297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:43.079864 14305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:43.080463 14303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:43.080504 14302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:43.080883 14297 server_base.cc:1034] running on GCE node
I20260928 10:51:43.081815 14297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:43.083566 14297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:43.084868 14297 hybrid_clock.cc:648] HybridClock initialized: now 1790592703084820 us; error 41 us; skew 500 ppm
I20260928 10:51:43.085397 14297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:43 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592703, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:43.089207 14297 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:51:43.090924 14297 webserver.cc:460] Webserver started at http://127.5.227.190:39085/ using document root <none> and password file <none>
I20260928 10:51:43.091645 14297 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:43.091830 14297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:43.092108 14297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:43.095417 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "34f70cba7e994e8397e5d22b697ff57a"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "b0531ab7a47015c03219d483b2687420"
server_key_iv: "62fdde55c461b8b1bd669891f640d239"
server_key_version: "encryptionkey@0"
I20260928 10:51:43.096338 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "34f70cba7e994e8397e5d22b697ff57a"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "b0531ab7a47015c03219d483b2687420"
server_key_iv: "62fdde55c461b8b1bd669891f640d239"
server_key_version: "encryptionkey@0"
I20260928 10:51:43.103507 14297 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 10:51:43.108382 14312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:43.110356 14297 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:51:43.110536 14297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "34f70cba7e994e8397e5d22b697ff57a"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "b0531ab7a47015c03219d483b2687420"
server_key_iv: "62fdde55c461b8b1bd669891f640d239"
server_key_version: "encryptionkey@0"
I20260928 10:51:43.110736 14297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:43.138991 14297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:43.146073 14297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:43.146516 14297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:43.172160 14297 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:33507
I20260928 10:51:43.172205 14363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:33507 every 8 connection(s)
I20260928 10:51:43.173686 14297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:43.175525 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 14297
I20260928 10:51:43.175802 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:51:43.176662 6030 external_mini_cluster.cc:1428] Setting key 9a79309d8e5a3fea1833fea998425e0a
I20260928 10:51:43.179268 14364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:51:43.196660 14364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Bootstrap starting.
Sep 28 10:51:43 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:43.200843 14364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Neither blocks nor log segments found. Creating new log.
I20260928 10:51:43.202250 14364 log.cc:824] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:43.206591 14364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: No bootstrap required, opened a new log
I20260928 10:51:43.211817 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:43.179826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.5.227.190:33507)
0928 10:51:43.181321 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:43.181334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:43.181381 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:51:43.184732 (+ 3351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:43.184767 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:43.185538 (+ 771us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:43.185996 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:43.188177 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:43.189647 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:43.191905 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:43.192776 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:43.192960 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:43.200345 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:43.200356 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:43.200378 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:43.200431 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:43.206023 (+ 5592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:43.207592 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:43.207601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:43.207609 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:43.207895 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:43.208376 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:43.208385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:43.208388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:43.209029 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:43.209399 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:43.209921 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 10:51:43.210577 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":574,"threads_started":1}
I20260928 10:51:43.217506 14364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:43.218006 14364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:51:43.218286 14364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f70cba7e994e8397e5d22b697ff57a, State: Initialized, Role: FOLLOWER
I20260928 10:51:43.218950 14364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:43.219187 14364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:43.219336 14364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:43.219530 14364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:51:43.222150 14364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:43.222759 14364 leader_election.cc:302] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f70cba7e994e8397e5d22b697ff57a; no voters:
I20260928 10:51:43.223587 14364 leader_election.cc:288] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:51:43.223832 14369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:51:43.224681 14369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 LEADER]: Becoming Leader. State: Replica: 34f70cba7e994e8397e5d22b697ff57a, State: Running, Role: LEADER
I20260928 10:51:43.225349 14369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:43.225871 14364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:43.228722 14370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f70cba7e994e8397e5d22b697ff57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } } }
I20260928 10:51:43.228701 14371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f70cba7e994e8397e5d22b697ff57a. Latest consensus state: current_term: 1 leader_uuid: "34f70cba7e994e8397e5d22b697ff57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } } }
I20260928 10:51:43.229183 14370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:43.229183 14371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:43.231949 14376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:43.238556 14376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:43.249857 14376 catalog_manager.cc:1347] Generated new cluster ID: 5c54fa2c848c48ab9dedb34311893251
I20260928 10:51:43.250020 14376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:43.283543 14376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:51:43.285200 14376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:43.297303 14376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Generated new TSK 0
I20260928 10:51:43.298456 14376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:51:43.373966 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:33507
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:43.740705 14390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:43.741155 14390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:43.741329 14390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:43.750170 14390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:43.750311 14390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:43.750490 14390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:43.762861 14390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:33507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:43.764303 14390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:43.765648 14390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:43.781713 14395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:43.781715 14396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:43.781831 14398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:43.782584 14390 server_base.cc:1034] running on GCE node
I20260928 10:51:43.783488 14390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:43.784763 14390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:43.786063 14390 hybrid_clock.cc:648] HybridClock initialized: now 1790592703785980 us; error 79 us; skew 500 ppm
I20260928 10:51:43.786541 14390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:43 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592703, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:43.795223 14390 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:51:43.797297 14390 webserver.cc:460] Webserver started at http://127.5.227.129:36069/ using document root <none> and password file <none>
I20260928 10:51:43.798115 14390 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:43.798278 14390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:43.798664 14390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:43.803496 14390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "81a19aaa7fba4bc1a5baa63541f204f0"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "1627e391ed8792bbeaffe3e2c48e9b32"
server_key_iv: "5644e9469adb1549a562a245afe00a24"
server_key_version: "encryptionkey@0"
I20260928 10:51:43.804898 14390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "81a19aaa7fba4bc1a5baa63541f204f0"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "1627e391ed8792bbeaffe3e2c48e9b32"
server_key_iv: "5644e9469adb1549a562a245afe00a24"
server_key_version: "encryptionkey@0"
I20260928 10:51:43.811362 14390 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:51:43.816466 14405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:43.818783 14390 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:51:43.818969 14390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "81a19aaa7fba4bc1a5baa63541f204f0"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "1627e391ed8792bbeaffe3e2c48e9b32"
server_key_iv: "5644e9469adb1549a562a245afe00a24"
server_key_version: "encryptionkey@0"
I20260928 10:51:43.819169 14390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:43.857411 14390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:43.863364 14390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:43.863706 14390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:43.864786 14390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:43.866230 14390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:43.866308 14390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:43.866400 14390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:43.866451 14390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:43.907014 14390 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:35453
I20260928 10:51:43.907050 14517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:35453 every 8 connection(s)
I20260928 10:51:43.908466 14390 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:43.915010 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 14390
I20260928 10:51:43.915696 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:51:43.917042 6030 external_mini_cluster.cc:1428] Setting key 3c0dc9bbc7adb891c0d5c9c8eea4b118
Sep 28 10:51:43 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592703, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:43.941272 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:43.912060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:49435 (local address 127.5.227.190:33507)
0928 10:51:43.912640 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:43.912645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:43.914333 (+ 1688us) server_negotiation.cc:407] Connection header received
0928 10:51:43.918567 (+ 4234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:43.918574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:43.918686 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:43.918835 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:43.921588 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:43.922452 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:43.925771 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:43.926219 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:43.926395 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:43.931965 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:43.931972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:43.931975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:43.932020 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:43.935513 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:43.937426 (+ 1913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:43.937433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:43.937436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:43.937628 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:43.938327 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:43.938331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:43.938334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:43.938629 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:43.938773 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:43.940403 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0928 10:51:43.940768 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":312,"threads_started":1}
I20260928 10:51:43.943056 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:43.912906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:49435)
0928 10:51:43.914094 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:43.914141 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:43.917941 (+ 3800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:43.919235 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:43.919245 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:43.920126 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:43.921243 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:43.921296 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:43.922757 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:43.922766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:43.925520 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:43.925559 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:43.926542 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:43.926561 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:43.926940 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:43.927949 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:43.927997 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:43.931507 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:43.935808 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:43.935823 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:43.935850 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:43.937159 (+ 1309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:43.937784 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:43.937792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:43.937799 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:43.938102 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:43.938787 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:43.938800 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:43.939846 (+ 1046us) client_negotiation.cc:769] Sending connection context
0928 10:51:43.940233 (+ 387us) client_negotiation.cc:240] Negotiation successful
0928 10:51:43.941153 (+ 920us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":704,"threads_started":1}
I20260928 10:51:43.945577 14518 heartbeater.cc:347] Connected to a master server at 127.5.227.190:33507
I20260928 10:51:43.946158 14518 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:51:43.947188 14518 heartbeater.cc:511] Master 127.5.227.190:33507 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:51:43.950560 14329 ts_manager.cc:194] Registered new tserver with Master: 81a19aaa7fba4bc1a5baa63541f204f0 (127.5.227.129:35453)
I20260928 10:51:43.952360 14329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:49435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:51:43.988376 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:33507
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:44.368294 14524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:44.368602 14524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:44.368701 14524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:44.376240 14524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:44.376355 14524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:44.376470 14524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:44.386714 14524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:33507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:44.388139 14524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:44.389484 14524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:44.404989 14530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:44.405035 14529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:44.405082 14532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:44.405933 14524 server_base.cc:1034] running on GCE node
I20260928 10:51:44.406876 14524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:44.407958 14524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:44.409246 14524 hybrid_clock.cc:648] HybridClock initialized: now 1790592704409173 us; error 63 us; skew 500 ppm
I20260928 10:51:44.409618 14524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:44 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:44.413130 14524 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:51:44.415342 14524 webserver.cc:460] Webserver started at http://127.5.227.130:35837/ using document root <none> and password file <none>
I20260928 10:51:44.416254 14524 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:44.416393 14524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:44.416806 14524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:44.420526 14524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "dfa71f86cb2146d5933ec2a92abe24ec"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "c4f1208f836812c5d8834f8023c6158d"
server_key_iv: "b11e5c8429f44d20d1c867df29872175"
server_key_version: "encryptionkey@0"
I20260928 10:51:44.421810 14524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "dfa71f86cb2146d5933ec2a92abe24ec"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "c4f1208f836812c5d8834f8023c6158d"
server_key_iv: "b11e5c8429f44d20d1c867df29872175"
server_key_version: "encryptionkey@0"
I20260928 10:51:44.429200 14524 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260928 10:51:44.433981 14539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:44.435861 14524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:51:44.436074 14524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "dfa71f86cb2146d5933ec2a92abe24ec"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "c4f1208f836812c5d8834f8023c6158d"
server_key_iv: "b11e5c8429f44d20d1c867df29872175"
server_key_version: "encryptionkey@0"
I20260928 10:51:44.436308 14524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:44.460727 14524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:44.466692 14524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:44.467057 14524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:44.468613 14524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:44.470144 14524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:44.470258 14524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:44.470412 14524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:44.470505 14524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:44.514628 14524 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:35485
I20260928 10:51:44.514725 14651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:35485 every 8 connection(s)
I20260928 10:51:44.516225 14524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:44.519456 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 14524
I20260928 10:51:44.519824 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:51:44.520399 6030 external_mini_cluster.cc:1428] Setting key eedb0aa5a94238eff2a965aa09ec3fa7
Sep 28 10:51:44 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:44.545692 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:44.519391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:33991 (local address 127.5.227.190:33507)
0928 10:51:44.520220 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:44.520228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:44.521447 (+ 1219us) server_negotiation.cc:407] Connection header received
0928 10:51:44.524358 (+ 2911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:44.524364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:44.524449 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:44.524543 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:44.527117 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:44.528157 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:44.531171 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:44.531997 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:44.532179 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:44.536807 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:44.536817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:44.536825 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:44.536878 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:44.541345 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:44.542459 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:44.542468 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:44.542474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:44.542667 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:44.543184 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:44.543193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:44.543198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:44.543640 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:44.543831 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:44.544769 (+ 938us) server_negotiation.cc:299] Negotiation successful
0928 10:51:44.545064 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":546,"threads_started":1}
I20260928 10:51:44.546424 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:44.520269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:33991)
0928 10:51:44.521250 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:44.521302 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:44.523972 (+ 2670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:44.524884 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:44.524902 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:44.525875 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:44.526862 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:44.526894 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:44.528357 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:44.528366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:44.530886 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:44.530908 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:44.532396 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:44.532403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:44.532735 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:44.533812 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:44.533839 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:44.536482 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:44.541611 (+ 5129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:44.541640 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:44.541656 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:44.542292 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:44.542819 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:44.542822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:44.542825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:44.543050 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:44.543910 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:44.543920 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:44.544344 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 10:51:44.544645 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 10:51:44.545067 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":388,"threads_started":1}
I20260928 10:51:44.548326 14652 heartbeater.cc:347] Connected to a master server at 127.5.227.190:33507
I20260928 10:51:44.548700 14652 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:51:44.549660 14652 heartbeater.cc:511] Master 127.5.227.190:33507 requested a full tablet report, sending...
I20260928 10:51:44.551407 14329 ts_manager.cc:194] Registered new tserver with Master: dfa71f86cb2146d5933ec2a92abe24ec (127.5.227.130:35485)
I20260928 10:51:44.552637 14329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:33991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:51:44.584394 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:33507
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:44.947145 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:44.947566 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:44.947788 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:44.956013 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:44.956255 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260928 10:51:44.956292 14518 heartbeater.cc:503] Master 127.5.227.190:33507 was elected leader, sending a full tablet report...
W20260928 10:51:44.956691 14658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:44.967572 14658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:33507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:44.969481 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:44.970878 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:44.986720 14663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:44.986778 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:44.986745 14666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:44.987183 14658 server_base.cc:1034] running on GCE node
I20260928 10:51:44.988471 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:44.989784 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:44.991051 14658 hybrid_clock.cc:648] HybridClock initialized: now 1790592704990985 us; error 50 us; skew 500 ppm
I20260928 10:51:44.991482 14658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:44 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:44.995352 14658 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:51:44.997447 14658 webserver.cc:460] Webserver started at http://127.5.227.131:44381/ using document root <none> and password file <none>
I20260928 10:51:44.998649 14658 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:44.998834 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:44.999225 14658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:51:45.002477 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "49ccd96b996c43faaa2f998a5b7c4c2f"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "ac77fcffc49047881a1a765f00dc2e68"
server_key_iv: "4ee0a00558f56bde687b5e70bf46a5ff"
server_key_version: "encryptionkey@0"
I20260928 10:51:45.003371 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "49ccd96b996c43faaa2f998a5b7c4c2f"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "ac77fcffc49047881a1a765f00dc2e68"
server_key_iv: "4ee0a00558f56bde687b5e70bf46a5ff"
server_key_version: "encryptionkey@0"
I20260928 10:51:45.010666 14658 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:51:45.014874 14673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:45.016723 14658 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:51:45.016925 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "49ccd96b996c43faaa2f998a5b7c4c2f"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "ac77fcffc49047881a1a765f00dc2e68"
server_key_iv: "4ee0a00558f56bde687b5e70bf46a5ff"
server_key_version: "encryptionkey@0"
I20260928 10:51:45.017149 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:45.040913 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:45.047088 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:45.047835 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:45.049168 14658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:45.050840 14658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:45.050959 14658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:45.051129 14658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:45.051218 14658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:45.093526 14658 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:38455
I20260928 10:51:45.093559 14785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:38455 every 8 connection(s)
I20260928 10:51:45.095026 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:45.101686 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 14658
I20260928 10:51:45.101930 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:51:45.102790 6030 external_mini_cluster.cc:1428] Setting key 865dd6d5eeba6da230305c752af60442
Sep 28 10:51:45 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:51:45.122874 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:45.098771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:56337 (local address 127.5.227.190:33507)
0928 10:51:45.099332 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:45.099339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:45.100978 (+ 1639us) server_negotiation.cc:407] Connection header received
0928 10:51:45.104477 (+ 3499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:45.104483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:45.104618 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:45.104753 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:45.106754 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.107524 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.109293 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.109706 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.109850 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:45.114561 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:45.114566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:45.114569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:45.114607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:45.118125 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.119818 (+ 1693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.119832 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.119836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.120002 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.120746 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.120752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.120761 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.121065 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:45.121241 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:45.122019 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 10:51:45.122449 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":263,"threads_started":1}
I20260928 10:51:45.124488 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:45.099299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:56337)
0928 10:51:45.100783 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:45.100825 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:45.103956 (+ 3131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:45.105038 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:45.105047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:45.105909 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:45.106547 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:45.106567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.107711 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.107715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:45.109111 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:45.109126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.109975 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.109982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:45.110247 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:45.111129 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:45.111189 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:45.114266 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:51:45.118510 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:45.118525 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:45.118542 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:45.119506 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:45.120268 (+ 762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:51:45.120274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:51:45.120280 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:51:45.120609 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:51:45.121210 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:51:45.121219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:51:45.121636 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 10:51:45.121924 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 10:51:45.122689 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":467,"threads_started":1}
I20260928 10:51:45.126744 14786 heartbeater.cc:347] Connected to a master server at 127.5.227.190:33507
I20260928 10:51:45.127308 14786 heartbeater.cc:464] Registering TS with master...
I20260928 10:51:45.128252 14786 heartbeater.cc:511] Master 127.5.227.190:33507 requested a full tablet report, sending...
I20260928 10:51:45.130017 14329 ts_manager.cc:194] Registered new tserver with Master: 49ccd96b996c43faaa2f998a5b7c4c2f (127.5.227.131:38455)
I20260928 10:51:45.130957 14329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:56337
I20260928 10:51:45.133893 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab.
Entry for principal oryx/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab.
WARNING: no policy specified for oryx/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab.
Entry for principal oryx/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab.
WARNING: no policy specified for oryx/127.5.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab.
Entry for principal oryx/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab.
I20260928 10:51:45.315295 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 14390
Sep 28 10:51:45 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:51:45.335592 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:45.316653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.5.227.129:35453)
0928 10:51:45.317778 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:45.317786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:45.317814 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:51:45.317875 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:45.317882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:45.318122 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:45.318269 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:45.319903 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.321321 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.323261 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.324184 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.324381 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:45.329066 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:45.329084 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:45.329102 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:45.329158 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:45.332469 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.333319 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.333327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.333330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.333463 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.333876 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.333881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.333884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.334287 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:45.334606 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:45.334927 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 10:51:45.335202 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":753,"threads_started":1}
I20260928 10:51:45.337132 14513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:45.423851 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 14390
I20260928 10:51:45.441488 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 14524
Sep 28 10:51:45 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:51:45.461782 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:45.442444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.5.227.130:35485)
0928 10:51:45.443055 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:45.443062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:45.443087 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:51:45.443152 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:45.443158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:45.443364 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:45.443512 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:45.445024 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.446293 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.447882 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.448793 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.448908 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:45.454070 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:45.454099 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:45.454117 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:45.454169 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:45.458136 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.459114 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.459121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.459125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.459269 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.459888 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.459893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.459895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.460321 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:45.460590 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:45.461051 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 10:51:45.461438 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
I20260928 10:51:45.463317 14647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:45.551981 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 14524
I20260928 10:51:45.571103 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 14658
Sep 28 10:51:45 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:51:45.595721 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:45.573000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.5.227.131:38455)
0928 10:51:45.573808 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:45.573819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:45.573858 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:51:45.573938 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:45.573949 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:45.574249 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:45.574574 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:45.576774 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.578523 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.581505 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:45.582304 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:45.582454 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:45.587971 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:45.587988 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:45.588006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:45.588068 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:45.591648 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.593075 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.593085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.593087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.593363 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:45.593898 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:45.593903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:45.593905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:45.594350 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:45.594650 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:45.595006 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 10:51:45.595245 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":449,"threads_started":1}
I20260928 10:51:45.597550 14781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:45.684927 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 14658
I20260928 10:51:45.701838 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 14297
I20260928 10:51:45.703696 14359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:51:45.812971 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 14297
I20260928 10:51:45.834506 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:33507
--webserver_interface=127.5.227.190
--webserver_port=39085
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:33507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:51:46.197292 14804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:46.197721 14804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:46.197919 14804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:46.206862 14804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:51:46.207098 14804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:46.207195 14804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:46.207283 14804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:51:46.207350 14804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:51:46.217657 14804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:33507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:33507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=39085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:46.219218 14804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:46.220634 14804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:46.234938 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:46.235114 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:46.234885 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:46.235643 14804 server_base.cc:1034] running on GCE node
I20260928 10:51:46.236655 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:46.238390 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:46.239696 14804 hybrid_clock.cc:648] HybridClock initialized: now 1790592706239636 us; error 52 us; skew 500 ppm
I20260928 10:51:46.240140 14804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:46 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:46.244272 14804 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:51:46.246054 14804 webserver.cc:460] Webserver started at http://127.5.227.190:39085/ using document root <none> and password file <none>
I20260928 10:51:46.246838 14804 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:46.247020 14804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:46.253799 14804 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:51:46.258169 14819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:46.260152 14804 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:51:46.260382 14804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "34f70cba7e994e8397e5d22b697ff57a"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "b0531ab7a47015c03219d483b2687420"
server_key_iv: "62fdde55c461b8b1bd669891f640d239"
server_key_version: "encryptionkey@0"
I20260928 10:51:46.261415 14804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:46.296463 14804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:46.301743 14804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:46.302049 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:46.324070 14804 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:33507
I20260928 10:51:46.324107 14870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:33507 every 8 connection(s)
I20260928 10:51:46.325666 14804 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:51:46.332171 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 14804
I20260928 10:51:46.333149 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:35453
--local_ip_for_outbound_sockets=127.5.227.129
--tserver_master_addrs=127.5.227.190:33507
--webserver_port=36069
--webserver_interface=127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:51:46.334319 14871 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:51:46.339915 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Bootstrap starting.
I20260928 10:51:46.361773 14871 log.cc:824] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:51:46.372131 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:51:46.372725 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Bootstrap complete.
I20260928 10:51:46.382093 14871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:46.383052 14871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f70cba7e994e8397e5d22b697ff57a, State: Initialized, Role: FOLLOWER
I20260928 10:51:46.383857 14871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:46.384109 14871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:51:46.384227 14871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:51:46.384436 14871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:51:46.388585 14871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:46.389215 14871 leader_election.cc:302] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f70cba7e994e8397e5d22b697ff57a; no voters:
I20260928 10:51:46.389950 14871 leader_election.cc:288] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:51:46.390210 14875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:51:46.391893 14875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [term 2 LEADER]: Becoming Leader. State: Replica: 34f70cba7e994e8397e5d22b697ff57a, State: Running, Role: LEADER
I20260928 10:51:46.392970 14875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } }
I20260928 10:51:46.393154 14871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:51:46.396127 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34f70cba7e994e8397e5d22b697ff57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } } }
I20260928 10:51:46.396565 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:46.397195 14877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f70cba7e994e8397e5d22b697ff57a. Latest consensus state: current_term: 2 leader_uuid: "34f70cba7e994e8397e5d22b697ff57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f70cba7e994e8397e5d22b697ff57a" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33507 } } }
I20260928 10:51:46.397508 14877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a [sys.catalog]: This master's current role is: LEADER
I20260928 10:51:46.408008 14882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:51:46.415007 14882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:51:46.416711 14882 catalog_manager.cc:1259] Loaded cluster ID: 5c54fa2c848c48ab9dedb34311893251
I20260928 10:51:46.416917 14882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:51:46.421825 14882 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:51:46.423662 14882 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34f70cba7e994e8397e5d22b697ff57a: Loaded TSK: 0
I20260928 10:51:46.425541 14882 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:51:46.714214 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:46.714532 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:46.714668 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:46.723295 14873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:46.723434 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:46.723651 14873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:51:46.734246 14873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.129.keytab
--principal=oryx/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:35453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=36069
--tserver_master_addrs=127.5.227.190:33507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:46.735782 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:46.737282 14873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:46.752193 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:46.755386 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:46.755410 14898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:46.758277 14873 server_base.cc:1034] running on GCE node
I20260928 10:51:46.759191 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:46.760587 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:46.761907 14873 hybrid_clock.cc:648] HybridClock initialized: now 1790592706761835 us; error 69 us; skew 500 ppm
I20260928 10:51:46.762357 14873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:46 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592706, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:46.767184 14873 init.cc:377] Logged in from keytab as oryx/127.5.227.129@KRBTEST.COM (short username oryx)
I20260928 10:51:46.769254 14873 webserver.cc:460] Webserver started at http://127.5.227.129:36069/ using document root <none> and password file <none>
I20260928 10:51:46.770133 14873 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:46.770310 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:46.777387 14873 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:51:46.781441 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:46.783378 14873 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:51:46.783619 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "81a19aaa7fba4bc1a5baa63541f204f0"
format_stamp: "Formatted at 2026-09-28 10:51:43 on dist-test-slave-8w8g"
server_key: "1627e391ed8792bbeaffe3e2c48e9b32"
server_key_iv: "5644e9469adb1549a562a245afe00a24"
server_key_version: "encryptionkey@0"
I20260928 10:51:46.784713 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:46.801662 14873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:46.807618 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:46.808269 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:46.809603 14873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:46.811319 14873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:46.811461 14873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:46.811614 14873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:46.811718 14873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:46.849771 14873 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:35453
I20260928 10:51:46.849869 15019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:35453 every 8 connection(s)
I20260928 10:51:46.851346 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:51:46.852311 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 14873
I20260928 10:51:46.853010 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:35485
--local_ip_for_outbound_sockets=127.5.227.130
--tserver_master_addrs=127.5.227.190:33507
--webserver_port=35837
--webserver_interface=127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:51:46 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:46 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:51:47.240041 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:47.240358 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:47.240510 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:47.250934 15024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:47.251096 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:47.251327 15024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:51:47.265373 15024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.130.keytab
--principal=oryx/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:35485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=35837
--tserver_master_addrs=127.5.227.190:33507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:47.266906 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:47.268389 15024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:47.290753 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:47.291316 15024 server_base.cc:1034] running on GCE node
W20260928 10:51:47.291841 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:47.291944 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:47.292768 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:47.293978 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:47.297430 15024 hybrid_clock.cc:648] HybridClock initialized: now 1790592707297341 us; error 84 us; skew 500 ppm
I20260928 10:51:47.297851 15024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:47 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592707, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:47.303244 15024 init.cc:377] Logged in from keytab as oryx/127.5.227.130@KRBTEST.COM (short username oryx)
I20260928 10:51:47.305300 15024 webserver.cc:460] Webserver started at http://127.5.227.130:35837/ using document root <none> and password file <none>
I20260928 10:51:47.306109 15024 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:47.306288 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:47.312871 15024 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:51:47.316633 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:47.318388 15024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:51:47.318605 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "dfa71f86cb2146d5933ec2a92abe24ec"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "c4f1208f836812c5d8834f8023c6158d"
server_key_iv: "b11e5c8429f44d20d1c867df29872175"
server_key_version: "encryptionkey@0"
I20260928 10:51:47.319893 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:47.344607 15024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:47.351148 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:47.351492 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:47.352751 15024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:47.354842 15024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:47.355150 15024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:47.355301 15024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:47.355397 15024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:47.407241 15024 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:35485
I20260928 10:51:47.407461 15152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:35485 every 8 connection(s)
I20260928 10:51:47.409065 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:51:47.416728 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15024
I20260928 10:51:47.417546 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:38455
--local_ip_for_outbound_sockets=127.5.227.131
--tserver_master_addrs=127.5.227.190:33507
--webserver_port=44381
--webserver_interface=127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:51:47 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:47 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:51:48.284425 15023 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.412s user 0.000s sys 0.014s
I20260928 10:51:48.286882 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:46.855426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:48773)
0928 10:51:46.860574 (+ 5148us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:46.860628 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:46.862439 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:46.864385 (+ 1946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:46.864394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:46.865073 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:46.865695 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:46.865715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:46.868368 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:46.868373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:46.869802 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:46.869818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:46.871483 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:46.871586 (+ 103us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:46.871938 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:46.872761 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:46.872790 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:48.285514 (+1412724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4805,"thread_start_us":4594,"threads_started":1}
W20260928 10:51:48.287122 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:46.855229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:48773 (local address 127.5.227.190:33507)
0928 10:51:46.856262 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:46.856281 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:46.860809 (+ 4528us) server_negotiation.cc:407] Connection header received
0928 10:51:46.863151 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:46.863172 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:46.863687 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:46.864159 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:46.865983 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:46.868132 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:46.870019 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:46.871196 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:46.871414 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.286104 (+1414690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:48773: BlockingRecv error: recv got EOF from 127.5.227.129:48773 (error 108)
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":429,"threads_started":1}
W20260928 10:51:48.288645 15020 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:33507 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.227.190:33507: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:51:48.309587 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:48.290412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:59331 (local address 127.5.227.190:33507)
0928 10:51:48.290635 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:48.290643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:48.291510 (+ 867us) server_negotiation.cc:407] Connection header received
0928 10:51:48.291585 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:48.291593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:48.291687 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:48.291815 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:48.296184 (+ 4369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.297515 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.302386 (+ 4871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.303305 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.303477 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.309353 (+ 5876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:59331: BlockingRecv error: recv got EOF from 127.5.227.129:59331 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:51:48.309607 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:48.289987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:59331)
0928 10:51:48.291173 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:48.291196 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:48.291358 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:48.293182 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:48.293190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:48.293845 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:48.294328 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.294350 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.297946 (+ 3596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.297952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.299894 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.299913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.303869 (+ 3956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.303880 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.304235 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.305064 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:48.305089 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:48.309117 (+ 4028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":14916,"threads_started":1}
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:48.319370 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:48.310424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:59545)
0928 10:51:48.310683 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:48.310703 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:48.310871 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:48.311217 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:48.311221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:48.311577 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:48.311825 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.311834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.312898 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.312903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.313928 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.313939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.314663 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.314669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.314850 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.315504 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:48.315522 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:48.319064 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165}
W20260928 10:51:48.319672 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:48.310487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:59545 (local address 127.5.227.190:33507)
0928 10:51:48.310665 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:48.310671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:48.310789 (+ 118us) server_negotiation.cc:407] Connection header received
0928 10:51:48.310999 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:48.311005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:48.311078 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:48.311188 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:48.311970 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.312740 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.314088 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.314493 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.314630 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.319299 (+ 4669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:59545: BlockingRecv error: recv got EOF from 127.5.227.129:59545 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 10:51:48.319977 15020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:51:48.325001 15157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:51:48.325626 15157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:51:48.325874 15157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:51:48.348141 15157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:51:48.348405 15157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:51:48.348675 15157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:51:48.376128 15157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/krb5kdc/oryx_127.5.227.131.keytab
--principal=oryx/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:38455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=44381
--tserver_master_addrs=127.5.227.190:33507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:51:48.377779 15157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:51:48.379216 15157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:51:48.397581 15165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:51:48.397590 15167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:48.398569 15157 server_base.cc:1034] running on GCE node
W20260928 10:51:48.398587 15164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:51:48.399614 15157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:51:48.400848 15157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:51:48.402117 15157 hybrid_clock.cc:648] HybridClock initialized: now 1790592708402067 us; error 116 us; skew 500 ppm
I20260928 10:51:48.402516 15157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592708, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:51:48.407773 15157 init.cc:377] Logged in from keytab as oryx/127.5.227.131@KRBTEST.COM (short username oryx)
I20260928 10:51:48.410624 15157 webserver.cc:460] Webserver started at http://127.5.227.131:44381/ using document root <none> and password file <none>
I20260928 10:51:48.412343 15157 fs_manager.cc:359] Metadata directory not provided
I20260928 10:51:48.412794 15157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:51:48.426321 15157 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.004s
I20260928 10:51:48.431805 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:48.435230 15157 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260928 10:51:48.435566 15157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "49ccd96b996c43faaa2f998a5b7c4c2f"
format_stamp: "Formatted at 2026-09-28 10:51:44 on dist-test-slave-8w8g"
server_key: "ac77fcffc49047881a1a765f00dc2e68"
server_key_iv: "4ee0a00558f56bde687b5e70bf46a5ff"
server_key_version: "encryptionkey@0"
I20260928 10:51:48.437891 15157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:51:48.473994 15157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:51:48.481987 15157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:51:48.482383 15157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:51:48.483603 15157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:51:48.485186 15157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:51:48.485316 15157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:48.485466 15157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:51:48.485555 15157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:51:48.536104 15157 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:38455
I20260928 10:51:48.536149 15286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:38455 every 8 connection(s)
I20260928 10:51:48.537703 15157 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:51:48.545194 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15157
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:48.580233 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:48.547141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.5.227.190:33507)
0928 10:51:48.551132 (+ 3991us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:48.551138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:48.551155 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:51:48.551206 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:48.551212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:48.551309 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:48.551430 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:48.556972 (+ 5542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.558792 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.561412 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.562466 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.562702 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.567245 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:51:48.567264 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:51:48.567305 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:51:48.567406 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:51:48.573872 (+ 6466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:48.575396 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:48.575411 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:48.575426 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:48.575829 (+ 403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:51:48.576446 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:51:48.576463 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:51:48.576469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:51:48.577470 (+ 1001us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:51:48.577960 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0928 10:51:48.578651 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 10:51:48.579548 (+ 897us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3850,"thread_start_us":3969,"threads_started":1}
W20260928 10:51:48.865196 15156 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.423s user 0.010s sys 0.020s
W20260928 10:51:48.867059 15155 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:47.414266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:34163 (local address 127.5.227.190:33507)
0928 10:51:47.414861 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:47.414866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:47.417310 (+ 2444us) server_negotiation.cc:407] Connection header received
0928 10:51:47.421323 (+ 4013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:47.421336 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:47.421496 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:47.421900 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:47.428577 (+ 6677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:47.431093 (+ 2516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:47.437339 (+ 6246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:47.438859 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:47.439349 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.866763 (+1427414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:34163: BlockingRecv error: recv got EOF from 127.5.227.130:34163 (error 108)
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":352,"threads_started":1}
I20260928 10:51:48.868240 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:47.415664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:34163)
0928 10:51:47.417065 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:47.417151 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:47.420609 (+ 3458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:47.423518 (+ 2909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:47.423544 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:47.425898 (+ 2354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:47.427922 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:47.428063 (+ 141us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:47.432360 (+ 4297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:47.432371 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:47.436623 (+ 4252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:47.436667 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:47.439482 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:47.439494 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:47.440231 (+ 737us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:47.441804 (+ 1573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:47.441846 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:48.866518 (+1424672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":488,"threads_started":1}
W20260928 10:51:48.869427 15153 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:33507 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.227.190:33507: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:48.890166 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:48.870323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:59163)
0928 10:51:48.870524 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:48.870564 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:48.870725 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:48.871297 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:48.871304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:48.871802 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:48.872048 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.872058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.874529 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.874533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.875885 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.875902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.877487 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.877493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.877637 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.884378 (+ 6741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:48.884411 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:48.889939 (+ 5528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 10:51:48.890682 15155 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:48.870474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:59163 (local address 127.5.227.190:33507)
0928 10:51:48.870675 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:48.870680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:48.870701 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:51:48.870965 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:48.870974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:48.871095 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:48.871261 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:48.872243 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.874279 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.876102 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.877209 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.877415 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.890325 (+ 12910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:59163: BlockingRecv error: recv got EOF from 127.5.227.130:59163 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:48 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:48.904047 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:48.891031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:37581)
0928 10:51:48.891219 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:48.891245 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:48.891381 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:48.892180 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:48.892184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:48.892507 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:48.892676 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.892685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.894948 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.894954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.896091 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.896105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.897863 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.897868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.898028 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.899228 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:48.899252 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:48.903816 (+ 4564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 10:51:48.904413 15153 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:51:48.904865 15155 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:48.891384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:37581 (local address 127.5.227.190:33507)
0928 10:51:48.891711 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:48.891721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:48.891773 (+ 52us) server_negotiation.cc:407] Connection header received
0928 10:51:48.891859 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:48.891867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:48.891985 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:48.892136 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:48.892870 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.894674 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.896260 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.897599 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.897811 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.904444 (+ 6633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:37581: BlockingRecv error: recv got EOF from 127.5.227.130:37581 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:49.338451 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:49.320828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:58151)
0928 10:51:49.321485 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:49.321506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:49.321686 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:49.322065 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:49.322069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:49.322592 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:49.322740 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.322748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.323967 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.323977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.326378 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.326407 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.327714 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.327725 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.328017 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.329276 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:49.329316 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:49.337865 (+ 8549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":464,"threads_started":1}
W20260928 10:51:49.338486 15155 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:49.321010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:58151 (local address 127.5.227.190:33507)
0928 10:51:49.321290 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:49.321302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:49.321578 (+ 276us) server_negotiation.cc:407] Connection header received
0928 10:51:49.321819 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:49.321824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:49.321924 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:49.322035 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:49.322885 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.323759 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.326668 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.327366 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.327632 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.338271 (+ 10639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:58151: BlockingRecv error: recv got EOF from 127.5.227.129:58151 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 10:51:49.681263 15289 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.129s user 0.000s sys 0.017s
W20260928 10:51:49.682523 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:48.540766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:50093 (local address 127.5.227.190:33507)
0928 10:51:48.541235 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:48.541246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:48.542242 (+ 996us) server_negotiation.cc:407] Connection header received
0928 10:51:48.544268 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:48.544279 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:48.544420 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:48.544603 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:48.547053 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.548141 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.549868 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.550417 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.550565 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.682311 (+1131746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:50093: BlockingRecv error: recv got EOF from 127.5.227.131:50093 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:51:49.683295 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:48.541204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:50093)
0928 10:51:48.542043 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:48.542092 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:48.543915 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:48.545048 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:48.545062 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:48.545911 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:48.546740 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.546776 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.548322 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.548332 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.549689 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:48.549703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:48.550678 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:48.550685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:48.551054 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:48.552242 (+ 1188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:48.552273 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:49.682024 (+1129751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":359,"threads_started":1}
W20260928 10:51:49.684719 15287 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:33507 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.227.190:33507: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:49.696013 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:49.685515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:38431)
0928 10:51:49.685696 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:49.685714 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:49.685814 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:49.686207 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:49.686211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:49.686637 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:49.686802 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.686820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.688111 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.688122 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.689422 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.689436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.690371 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.690377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.690539 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.691224 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:49.691242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:49.695683 (+ 4441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 10:51:49.696314 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:49.685621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:38431 (local address 127.5.227.190:33507)
0928 10:51:49.685838 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:49.685844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:49.685886 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:51:49.685951 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:49.685955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:49.686043 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:49.686147 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:49.687023 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.687902 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.689580 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.690169 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.690295 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.696042 (+ 5747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:38431: BlockingRecv error: recv got EOF from 127.5.227.131:38431 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:49.707288 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:49.697120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:46321)
0928 10:51:49.697282 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:49.697300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:49.697447 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:49.697784 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:49.697788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:49.698268 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:49.698513 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.698524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.699608 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.699614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.700671 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.700684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.701420 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.701427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.701628 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.702364 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:49.702382 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:49.707050 (+ 4668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 10:51:49.707662 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:49.697028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:46321 (local address 127.5.227.190:33507)
0928 10:51:49.697210 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:49.697216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:49.697374 (+ 158us) server_negotiation.cc:407] Connection header received
0928 10:51:49.697565 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:49.697569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:49.697629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:49.697753 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:49.698681 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.699462 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.700833 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.701256 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.701405 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.707337 (+ 5932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:46321: BlockingRecv error: recv got EOF from 127.5.227.131:46321 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 10:51:49.707695 15287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:49 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:49.916280 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:49.905209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:52171)
0928 10:51:49.906081 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:49.906101 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:49.906227 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:49.906614 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:49.906619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:49.907109 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:49.907277 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.907286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.908600 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.908605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.909662 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:49.909675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.910418 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.910422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:49.910597 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.911310 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:49.911331 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:49.915912 (+ 4581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":546,"threads_started":1}
W20260928 10:51:49.916680 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:49.905320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:52171 (local address 127.5.227.190:33507)
0928 10:51:49.905557 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:49.905567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:49.906201 (+ 634us) server_negotiation.cc:407] Connection header received
0928 10:51:49.906372 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:49.906377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:49.906453 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:49.906548 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:49.907450 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.908270 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.909806 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:49.910232 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:49.910343 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:49.916323 (+ 5980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:52171: BlockingRecv error: recv got EOF from 127.5.227.130:52171 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 10:51:50 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:50 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:50.351171 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:50.339637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:51497)
0928 10:51:50.340158 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:50.340174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:50.340300 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:50.340690 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:50.340694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:50.341098 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:50.341288 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:50.341297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.342776 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.342780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:50.343819 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:50.343833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.344713 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.344719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:50.344877 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:50.345682 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:50.345702 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:50.350813 (+ 5111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":292,"threads_started":1}
W20260928 10:51:50.351374 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:50.339697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:51497 (local address 127.5.227.190:33507)
0928 10:51:50.339893 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:50.339898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:50.340260 (+ 362us) server_negotiation.cc:407] Connection header received
0928 10:51:50.340453 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:50.340457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:50.340519 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:50.340603 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:50.341501 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.342598 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.344006 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.344515 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.344636 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:50.351068 (+ 6432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:51497: BlockingRecv error: recv got EOF from 127.5.227.129:51497 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 10:51:50 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:50 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:51:50.720432 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:50.708739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:50503 (local address 127.5.227.190:33507)
0928 10:51:50.708970 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:50.708977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:50.709407 (+ 430us) server_negotiation.cc:407] Connection header received
0928 10:51:50.709618 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:50.709624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:50.709703 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:50.709797 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:50.710670 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.711530 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.713414 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.713853 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.714033 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:50.720235 (+ 6202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:50503: BlockingRecv error: recv got EOF from 127.5.227.131:50503 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 10:51:50.720469 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:50.708551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:50503)
0928 10:51:50.709284 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:50.709303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:50.709446 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:50.709877 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:50.709881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:50.710351 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:50.710502 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:50.710511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.711775 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.711779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:50.713185 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:50.713206 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.714204 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.714210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:50.714340 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:50.715260 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:50.715303 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:50.719779 (+ 4476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":435,"threads_started":1}
Sep 28 10:51:50 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:50 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:50.928768 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:50.917608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:53487)
0928 10:51:50.918230 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:50.918250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:50.918407 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:50.918788 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:50.918900 (+ 112us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:50.919464 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:50.919721 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:50.919757 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.920934 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.920938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:50.921992 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:50.922006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.922875 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.922881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:50.923175 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:50.924134 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:50.924153 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:50.928503 (+ 4350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":351,"threads_started":1}
W20260928 10:51:50.929051 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:50.917746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:53487 (local address 127.5.227.190:33507)
0928 10:51:50.918008 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:50.918021 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:50.918387 (+ 366us) server_negotiation.cc:407] Connection header received
0928 10:51:50.918572 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:50.918577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:50.918649 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:50.918744 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:50.919929 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.920769 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.922211 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:50.922677 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:50.922795 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:50.928803 (+ 6008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:53487: BlockingRecv error: recv got EOF from 127.5.227.130:53487 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 10:51:51 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:51 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:51.362370 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:51.352602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:34033)
0928 10:51:51.353176 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:51.353192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:51.353351 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:51.353694 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:51.353698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:51.354080 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:51.354244 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:51.354253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.355316 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.355322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:51.356326 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:51.356338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.357138 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.357143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:51.357321 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:51.358047 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:51.358069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:51.362127 (+ 4058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":323,"threads_started":1}
W20260928 10:51:51.362782 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:51.352680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:34033 (local address 127.5.227.190:33507)
0928 10:51:51.352942 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:51.352950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:51.353284 (+ 334us) server_negotiation.cc:407] Connection header received
0928 10:51:51.353482 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:51.353489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:51.353562 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:51.353664 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:51.354397 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.355163 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.356476 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.356942 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.357109 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:51.362497 (+ 5388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:34033: BlockingRecv error: recv got EOF from 127.5.227.129:34033 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 10:51:51 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:51 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:51.733276 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:51.721635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:40463)
0928 10:51:51.722173 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:51.722189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:51.722312 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:51.722718 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:51.722722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:51.723168 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:51.723392 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:51.723402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.724890 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.724895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:51.726040 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:51.726055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.726950 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.726956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:51.727115 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:51.728125 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:51.728157 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:51.732852 (+ 4695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":302,"threads_started":1}
W20260928 10:51:51.733414 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:51.721721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:40463 (local address 127.5.227.190:33507)
0928 10:51:51.721932 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:51.721936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:51.722308 (+ 372us) server_negotiation.cc:407] Connection header received
0928 10:51:51.722405 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:51.722409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:51.722524 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:51.722624 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:51.723655 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.724695 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.726247 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.726734 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.726853 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:51.733107 (+ 6254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:40463: BlockingRecv error: recv got EOF from 127.5.227.131:40463 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 10:51:51 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:51 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:51.940325 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:51.930126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:56517)
0928 10:51:51.930700 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:51.930717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:51.930840 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:51.931171 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:51.931175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:51.931590 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:51.931826 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:51.931841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.933015 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.933019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:51.934195 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:51.934211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.935028 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.935033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:51.935175 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:51.936044 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:51.936064 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:51.939981 (+ 3917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":339,"threads_started":1}
W20260928 10:51:51.940649 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:51.930201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:56517 (local address 127.5.227.190:33507)
0928 10:51:51.930485 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:51.930494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:51.930813 (+ 319us) server_negotiation.cc:407] Connection header received
0928 10:51:51.930949 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:51.930954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:51.931019 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:51.931111 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:51.932081 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.932818 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.934371 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:51.934820 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:51.934943 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:51.940276 (+ 5333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:56517: BlockingRecv error: recv got EOF from 127.5.227.130:56517 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 10:51:52 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:52 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:52.375160 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:52.363494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:50229)
0928 10:51:52.364269 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:52.364288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:52.364454 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:52.364968 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:52.365119 (+ 151us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:52.365636 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:52.365826 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:52.365841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.367167 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.367173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:52.368714 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:52.368729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.369923 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.369933 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:52.370128 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:52.370997 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:52.371018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:52.374875 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":523,"threads_started":1}
W20260928 10:51:52.375761 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:52.363719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:50229 (local address 127.5.227.190:33507)
0928 10:51:52.364225 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:52.364231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:52.364430 (+ 199us) server_negotiation.cc:407] Connection header received
0928 10:51:52.364591 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:52.364595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:52.364674 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:52.364795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:52.366094 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.366964 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.368870 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.369650 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.369766 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:52.375484 (+ 5718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:50229: BlockingRecv error: recv got EOF from 127.5.227.129:50229 (error 108)
Metrics: {"server-negotiator.queue_time_us":301}
Sep 28 10:51:52 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:52 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:52.745599 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:52.734399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:60019)
0928 10:51:52.735105 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:52.735125 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:52.735286 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:52.735885 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:52.735899 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:52.736316 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:52.736492 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:52.736502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.737786 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.737795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:52.738722 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:52.738735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.739509 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.739516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:52.739700 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:52.740367 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:52.740387 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:52.745277 (+ 4890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":356,"threads_started":1}
W20260928 10:51:52.746214 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:52.734564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:60019 (local address 127.5.227.190:33507)
0928 10:51:52.734783 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:52.734789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:52.735234 (+ 445us) server_negotiation.cc:407] Connection header received
0928 10:51:52.735581 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:52.735585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:52.735654 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:52.735862 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:52.736661 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.737620 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.738894 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.739332 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.739480 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:52.745604 (+ 6124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:60019: BlockingRecv error: recv got EOF from 127.5.227.131:60019 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 10:51:52 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:52 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:52.952134 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:52.941583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:41835)
0928 10:51:52.942232 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:52.942250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:52.942423 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:52.942829 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:52.942835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:52.943321 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:52.943506 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:52.943516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.944739 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.944746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:52.945922 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:52.945936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.946939 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.946946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:52.947149 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:52.948009 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:52.948028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:52.951726 (+ 3698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":392,"threads_started":1}
W20260928 10:51:52.952419 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:52.941709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:41835 (local address 127.5.227.190:33507)
0928 10:51:52.941983 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:52.941992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:52.942352 (+ 360us) server_negotiation.cc:407] Connection header received
0928 10:51:52.942584 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:52.942590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:52.942668 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:52.942790 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:52.943693 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.944573 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.946097 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:52.946688 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:52.946902 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:52.952119 (+ 5217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:41835: BlockingRecv error: recv got EOF from 127.5.227.130:41835 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 10:51:53 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:53 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:53.387476 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:53.376350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:41045)
0928 10:51:53.377044 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:53.377061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:53.377243 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:53.377609 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:53.377613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:53.378151 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:53.378374 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:53.378384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.379484 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.379491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:53.380777 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:53.380792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.381525 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.381533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:53.381733 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:53.382445 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:53.382466 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:53.387177 (+ 4711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":321,"threads_started":1}
W20260928 10:51:53.387809 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:53.376608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:41045 (local address 127.5.227.190:33507)
0928 10:51:53.376898 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:53.376949 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:53.377161 (+ 212us) server_negotiation.cc:407] Connection header received
0928 10:51:53.377380 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:53.377385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:53.377470 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:53.377583 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:53.378518 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.379317 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.380908 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.381344 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.381552 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:53.387490 (+ 5938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:41045: BlockingRecv error: recv got EOF from 127.5.227.129:41045 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 10:51:53 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:53 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:53.758370 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:53.746993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:49905)
0928 10:51:53.747647 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:53.747664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:53.747819 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:53.748288 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:53.748292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:53.748810 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:53.749000 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:53.749010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.750292 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.750297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:53.751555 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:53.751570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.752629 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.752634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:53.752784 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:53.753552 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:53.753572 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:53.758021 (+ 4449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":403,"threads_started":1}
W20260928 10:51:53.758620 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:53.747230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:49905 (local address 127.5.227.190:33507)
0928 10:51:53.747503 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:53.747508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:53.747806 (+ 298us) server_negotiation.cc:407] Connection header received
0928 10:51:53.747995 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:53.747999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:53.748090 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:53.748188 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:53.749209 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.750091 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.751821 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.752383 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.752511 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:53.758368 (+ 5857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:49905: BlockingRecv error: recv got EOF from 127.5.227.131:49905 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 10:51:53 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:53 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:53.974767 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:53.953432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:37705)
0928 10:51:53.954038 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:53.954060 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:53.954201 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:53.954581 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:53.954585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:53.955054 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:53.955212 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:53.955221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.956368 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.956374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:53.957477 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:53.957491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.958258 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.958263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:53.958511 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:53.969643 (+ 11132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:53.969668 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:53.974402 (+ 4734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":322,"threads_started":1}
W20260928 10:51:53.975044 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:53.953839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:37705 (local address 127.5.227.190:33507)
0928 10:51:53.954160 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:53.954165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:53.954185 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:51:53.954362 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:53.954368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:53.954435 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:53.954559 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:53.955389 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.956208 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.957628 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:53.958099 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:53.958243 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:53.974654 (+ 16411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:37705: BlockingRecv error: recv got EOF from 127.5.227.130:37705 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Sep 28 10:51:54 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:54 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:54.398833 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:54.388768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:58699)
0928 10:51:54.389408 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:54.389425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:54.389594 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:54.389986 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:54.389991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:54.390529 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:54.390683 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:54.390692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.391719 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.391762 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:54.392698 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:54.392709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.393464 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.393471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:54.393657 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:54.394449 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:54.394467 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:54.398588 (+ 4121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":316,"threads_started":1}
W20260928 10:51:54.399101 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:54.389011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:58699 (local address 127.5.227.190:33507)
0928 10:51:54.389291 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:54.389297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:54.389525 (+ 228us) server_negotiation.cc:407] Connection header received
0928 10:51:54.389761 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:54.389765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:54.389829 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:54.389953 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:54.390854 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.391574 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.392866 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.393301 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.393447 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:54.398849 (+ 5402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:58699: BlockingRecv error: recv got EOF from 127.5.227.129:58699 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 10:51:54 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:54 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:54.769420 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:54.759621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:59547)
0928 10:51:54.760251 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:54.760273 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:54.760417 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:54.760751 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:54.760756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:54.761205 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:54.761349 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:54.761358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.762394 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.762400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:54.763347 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:54.763358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.764164 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.764171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:54.764356 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:54.765112 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:54.765138 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:54.769106 (+ 3968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":361,"threads_started":1}
W20260928 10:51:54.769842 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:54.759768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:59547 (local address 127.5.227.190:33507)
0928 10:51:54.760019 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:54.760025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:54.760368 (+ 343us) server_negotiation.cc:407] Connection header received
0928 10:51:54.760535 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:54.760541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:54.760609 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:54.760715 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:54.761503 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.762230 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.763502 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.764009 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.764151 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:54.769573 (+ 5422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:59547: BlockingRecv error: recv got EOF from 127.5.227.131:59547 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 10:51:54 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:54 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:54.987932 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:54.976107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:52239)
0928 10:51:54.976716 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:54.976735 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:54.976922 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:54.977361 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:54.977366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:54.977888 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:54.978118 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:54.978134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.979663 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.979669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:54.980791 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:54.980811 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.981628 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.981634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:54.981818 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:54.982458 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:54.982479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:54.987551 (+ 5072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":337,"threads_started":1}
W20260928 10:51:54.988292 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:54.976317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:52239 (local address 127.5.227.190:33507)
0928 10:51:54.976627 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:54.976636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:54.976837 (+ 201us) server_negotiation.cc:407] Connection header received
0928 10:51:54.977127 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:54.977141 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:54.977212 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:54.977333 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:54.978534 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.979500 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.980958 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:54.981462 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:54.981606 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:54.987990 (+ 6384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:52239: BlockingRecv error: recv got EOF from 127.5.227.130:52239 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 28 10:51:55 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:55 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:55.410729 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:55.400144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:57115)
0928 10:51:55.400891 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:55.400908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:55.401029 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:55.401493 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:55.401497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:55.401932 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:55.402122 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:55.402130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.403245 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.403250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:55.404412 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:55.404426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.405389 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.405394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:55.405523 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:55.406235 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:55.406255 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:55.410319 (+ 4064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":280,"threads_started":1}
W20260928 10:51:55.410856 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:55.400211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:57115 (local address 127.5.227.190:33507)
0928 10:51:55.400429 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:55.400436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:55.401012 (+ 576us) server_negotiation.cc:407] Connection header received
0928 10:51:55.401198 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:55.401203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:55.401320 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:55.401432 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:55.402284 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.403091 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.404587 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.405087 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.405327 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:55.410577 (+ 5250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:57115: BlockingRecv error: recv got EOF from 127.5.227.129:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 10:51:55 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:55 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:51:55.783186 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:55.770777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:53873 (local address 127.5.227.190:33507)
0928 10:51:55.771201 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:55.771208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:55.771621 (+ 413us) server_negotiation.cc:407] Connection header received
0928 10:51:55.771860 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:55.771865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:55.771939 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:55.772051 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:55.773129 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.774020 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.775662 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.776311 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.776441 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:55.783004 (+ 6563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:53873: BlockingRecv error: recv got EOF from 127.5.227.131:53873 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
I20260928 10:51:55.783170 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:55.770666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:53873)
0928 10:51:55.771418 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:55.771443 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:55.771625 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:55.772153 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:55.772157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:55.772687 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:55.772896 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:55.772906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.774281 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.774287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:55.775398 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:55.775413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.776533 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.776539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:55.776706 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:55.777448 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:55.777473 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:55.782722 (+ 5249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":465,"threads_started":1}
Sep 28 10:51:55 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:55 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:56.000526 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:55.989158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:59171)
0928 10:51:55.989877 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:55.989903 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:55.990088 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:55.990487 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:55.990492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:55.990981 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:55.991174 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:55.991184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.992407 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.992415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:55.993484 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:55.993497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.994414 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.994421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:55.994619 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:55.995397 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:55.995419 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:56.000160 (+ 4741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":275,"threads_started":1}
W20260928 10:51:56.000717 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:55.989224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:59171 (local address 127.5.227.190:33507)
0928 10:51:55.989456 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:55.989465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:55.990008 (+ 543us) server_negotiation.cc:407] Connection header received
0928 10:51:55.990233 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:55.990238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:55.990337 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:55.990460 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:55.991337 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.992258 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.993654 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:55.994233 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:55.994392 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:56.000459 (+ 6067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:59171: BlockingRecv error: recv got EOF from 127.5.227.130:59171 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 10:51:56 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:56 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:56.424057 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:56.411803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:49721)
0928 10:51:56.412537 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:56.412554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:56.412677 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:56.413110 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:56.413116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:56.413744 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:56.414000 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:56.414012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.415113 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.415119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:56.416349 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:56.416363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.417240 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.417246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:56.417383 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:56.418180 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:56.418201 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:56.423713 (+ 5512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":366,"threads_started":1}
W20260928 10:51:56.424500 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:56.411864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:49721 (local address 127.5.227.190:33507)
0928 10:51:56.412136 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:56.412147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:56.412679 (+ 532us) server_negotiation.cc:407] Connection header received
0928 10:51:56.412787 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:56.412792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:56.412873 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:56.412990 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:56.414188 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.414944 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.416533 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.417038 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.417165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:56.424154 (+ 6989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:49721: BlockingRecv error: recv got EOF from 127.5.227.129:49721 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 10:51:56 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:56 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:56.796592 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:56.784463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:58249)
0928 10:51:56.785244 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:56.785261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:56.785403 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:56.785796 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:56.785800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:56.786348 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:56.786525 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:56.786534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.787798 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.787804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:56.789187 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:56.789202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.790286 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.790291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:56.790414 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:56.791228 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:56.791250 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:56.796331 (+ 5081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":433,"threads_started":1}
W20260928 10:51:56.796803 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:56.784737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:58249 (local address 127.5.227.190:33507)
0928 10:51:56.785039 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:56.785052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:56.785389 (+ 337us) server_negotiation.cc:407] Connection header received
0928 10:51:56.785521 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:56.785525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:56.785593 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:56.785690 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:56.786714 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.787580 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.789511 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:56.790023 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:56.790150 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:56.796561 (+ 6411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:58249: BlockingRecv error: recv got EOF from 127.5.227.131:58249 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 10:51:57 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:57 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:57.012491 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:57.002044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:49311)
0928 10:51:57.002797 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:57.002814 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:57.002958 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:57.003308 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:57.003314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:57.003890 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:57.004070 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:57.004079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.005519 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.005524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:57.006555 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:57.006570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.007503 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.007509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:57.007637 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:57.008488 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:57.008509 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:57.012222 (+ 3713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":458,"threads_started":1}
W20260928 10:51:57.012982 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:57.002229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:49311 (local address 127.5.227.190:33507)
0928 10:51:57.002509 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:57.002519 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:57.002920 (+ 401us) server_negotiation.cc:407] Connection header received
0928 10:51:57.003093 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:57.003097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:57.003159 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:57.003242 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:57.004239 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.005287 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.006771 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.007267 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.007426 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:57.012494 (+ 5068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:49311: BlockingRecv error: recv got EOF from 127.5.227.130:49311 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 10:51:57 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:57 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:57.436875 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:57.425524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:36549)
0928 10:51:57.426045 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:57.426063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:57.426205 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:57.426622 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:57.426625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:57.427130 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:57.427334 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:57.427344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.428774 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.428779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:57.430498 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:57.430515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.431430 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.431436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:57.431583 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:57.432382 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:57.432403 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:57.436591 (+ 4188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
W20260928 10:51:57.437163 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:57.425606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:36549 (local address 127.5.227.190:33507)
0928 10:51:57.425886 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:57.425892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:57.426245 (+ 353us) server_negotiation.cc:407] Connection header received
0928 10:51:57.426367 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:57.426372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:57.426464 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:57.426561 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:57.427508 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.428602 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.430664 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.431185 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.431303 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:57.436881 (+ 5578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:36549: BlockingRecv error: recv got EOF from 127.5.227.129:36549 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 28 10:51:57 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:57 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:57.811802 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:57.797942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:53171)
0928 10:51:57.798590 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:57.798617 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:57.798775 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:57.799252 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:57.799258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:57.799999 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:57.800233 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:57.800245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.801612 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.801616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:57.803530 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:57.803574 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.804945 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.804954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:57.805189 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:57.806239 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:57.806260 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:57.811349 (+ 5089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":346,"threads_started":1}
W20260928 10:51:57.811892 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:57.798091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:53171 (local address 127.5.227.190:33507)
0928 10:51:57.798249 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:57.798254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:57.798712 (+ 458us) server_negotiation.cc:407] Connection header received
0928 10:51:57.798989 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:57.798993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:57.799062 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:57.799224 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:57.800444 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.801387 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.803855 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:57.804697 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:57.804863 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:57.811590 (+ 6727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:53171: BlockingRecv error: recv got EOF from 127.5.227.131:53171 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 10:51:58 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:58 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:58.024499 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:58.013590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:49903)
0928 10:51:58.014134 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:58.014149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:58.014312 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:58.014684 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:58.014689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:58.015137 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:58.015301 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:58.015310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.016440 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.016447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:58.017423 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:58.017436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.018221 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.018227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:58.018408 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:58.019196 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:58.019217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:58.023965 (+ 4748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
W20260928 10:51:58.024621 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:58.013686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:49903 (local address 127.5.227.190:33507)
0928 10:51:58.013984 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:58.013991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:58.014238 (+ 247us) server_negotiation.cc:407] Connection header received
0928 10:51:58.014456 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:58.014463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:58.014530 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:58.014635 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:58.015494 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.016291 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.017583 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.018054 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.018200 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:58.024177 (+ 5977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:49903: BlockingRecv error: recv got EOF from 127.5.227.130:49903 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 10:51:58 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:58 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:58.448879 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:58.437998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:55877)
0928 10:51:58.438639 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:58.438657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:58.438821 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:58.439270 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:58.439274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:58.439656 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:58.440031 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:58.440042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.441286 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.441290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:58.442650 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:58.442663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.443447 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.443452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:58.443578 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:58.444357 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:58.444377 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:58.448639 (+ 4262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":377,"threads_started":1}
W20260928 10:51:58.449223 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:58.438174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:55877 (local address 127.5.227.190:33507)
0928 10:51:58.438365 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:58.438370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:58.438739 (+ 369us) server_negotiation.cc:407] Connection header received
0928 10:51:58.438978 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:58.438982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:58.439045 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:58.439141 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:58.440219 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.441113 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.442830 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.443257 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.443409 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:58.448867 (+ 5458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:55877: BlockingRecv error: recv got EOF from 127.5.227.129:55877 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 10:51:58 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:58 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:58.824292 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:58.813456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:44717)
0928 10:51:58.814040 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:58.814056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:58.814229 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:58.814648 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:58.814652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:58.815140 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:58.815285 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:58.815324 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.816423 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.816429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:58.817392 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:58.817404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.818123 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.818130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:58.818325 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:58.819036 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:58.819056 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:58.824049 (+ 4993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":311,"threads_started":1}
W20260928 10:51:58.824504 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:58.813555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:44717 (local address 127.5.227.190:33507)
0928 10:51:58.813893 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:58.813900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:58.814154 (+ 254us) server_negotiation.cc:407] Connection header received
0928 10:51:58.814435 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:58.814441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:58.814510 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:58.814621 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:58.815470 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.816285 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.817551 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:58.817960 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:58.818090 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:58.824234 (+ 6144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:44717: BlockingRecv error: recv got EOF from 127.5.227.131:44717 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 10:51:59 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:59 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:59.041906 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:59.025918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:58195)
0928 10:51:59.026619 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:59.026638 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:59.026919 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:59.027306 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:59.027310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:59.027691 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:59.028044 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:59.028054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.029505 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.029510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:59.030516 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:59.030530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.031545 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.031551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:59.031677 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:59.037015 (+ 5338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:59.037039 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:59.041479 (+ 4440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":441,"threads_started":1}
W20260928 10:51:59.041986 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:59.026031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:58195 (local address 127.5.227.190:33507)
0928 10:51:59.026394 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:59.026399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:59.026946 (+ 547us) server_negotiation.cc:407] Connection header received
0928 10:51:59.027049 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:59.027053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:59.027115 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:59.027203 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:59.028237 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.029262 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.030737 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.031242 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.031369 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:59.041797 (+ 10428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:58195: BlockingRecv error: recv got EOF from 127.5.227.130:58195 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 10:51:59 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:59 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:59.460220 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:59.450103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:57967)
0928 10:51:59.450679 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:59.450695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:59.450818 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:59.451182 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:59.451186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:59.451603 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:59.451851 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:59.451867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.453035 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.453040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:59.453981 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:59.453993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.455084 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.455089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:59.455253 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:59.456107 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:59.456127 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:59.459984 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":333,"threads_started":1}
W20260928 10:51:59.460500 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:59.450311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:57967 (local address 127.5.227.190:33507)
0928 10:51:59.450563 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:59.450570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:59.450807 (+ 237us) server_negotiation.cc:407] Connection header received
0928 10:51:59.450928 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:59.450931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:59.450995 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:59.451085 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:59.452037 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.452852 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.454340 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.454828 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.454951 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:59.460200 (+ 5249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:57967: BlockingRecv error: recv got EOF from 127.5.227.129:57967 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 28 10:51:59 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:51:59 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:51:59.836382 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:51:59.825568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:54245)
0928 10:51:59.826281 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 10:51:59.826298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:51:59.826481 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:51:59.826989 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:51:59.826995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:51:59.827540 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:51:59.827794 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:59.827807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.829084 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.829090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:59.830208 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:51:59.830222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.831203 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.831210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:51:59.831416 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:59.832176 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:51:59.832196 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:51:59.836122 (+ 3926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":444,"threads_started":1}
W20260928 10:51:59.836714 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:51:59.825660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:54245 (local address 127.5.227.190:33507)
0928 10:51:59.826072 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 10:51:59.826082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:51:59.826401 (+ 319us) server_negotiation.cc:407] Connection header received
0928 10:51:59.826616 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:51:59.826622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:51:59.826698 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:51:59.826965 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:51:59.827960 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.828921 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.830528 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:51:59.831007 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:51:59.831183 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:51:59.836409 (+ 5226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:54245: BlockingRecv error: recv got EOF from 127.5.227.131:54245 (error 108)
Metrics: {"server-negotiator.queue_time_us":231}
Sep 28 10:52:00 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:00 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:00.056994 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:00.043372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:33733)
0928 10:52:00.044082 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:00.044100 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:00.044238 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:00.044616 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:00.044620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:00.045127 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:00.045439 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:00.045450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.046820 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.046825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:00.048058 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:00.048074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.049177 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.049182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:00.049345 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:00.050368 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:00.050397 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:00.056612 (+ 6215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":295,"threads_started":1}
W20260928 10:52:00.057193 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:00.043548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:33733 (local address 127.5.227.190:33507)
0928 10:52:00.043980 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:00.043985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:00.044215 (+ 230us) server_negotiation.cc:407] Connection header received
0928 10:52:00.044380 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:00.044384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:00.044449 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:00.044540 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:00.045632 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.046577 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.048303 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.048939 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.049084 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:00.056876 (+ 7792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:33733: BlockingRecv error: recv got EOF from 127.5.227.130:33733 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 10:52:00 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:00 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:00.472966 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:00.461461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:52915)
0928 10:52:00.462068 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:00.462085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:00.462256 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:00.462629 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:00.462634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:00.463093 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:00.463310 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:00.463322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.465151 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.465156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:00.466527 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:00.466547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.467564 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.467570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:00.467920 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:00.468658 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:00.468684 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:00.472577 (+ 3893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":330,"threads_started":1}
W20260928 10:52:00.473055 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:00.461564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:52915 (local address 127.5.227.190:33507)
0928 10:52:00.461849 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:00.461857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:00.462179 (+ 322us) server_negotiation.cc:407] Connection header received
0928 10:52:00.462399 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:00.462404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:00.462479 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:00.462590 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:00.463581 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.464942 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.466898 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.467361 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.467491 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:00.472787 (+ 5296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:52915: BlockingRecv error: recv got EOF from 127.5.227.129:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 28 10:52:00 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:00 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:00.848523 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:00.837726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:52667)
0928 10:52:00.838352 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:00.838368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:00.838551 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:00.839010 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:00.839015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:00.839554 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:00.839890 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:00.839906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.841223 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.841228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:00.842436 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:00.842451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.843279 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.843285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:00.843466 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:00.844144 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:00.844164 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:00.848257 (+ 4093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":325,"threads_started":1}
W20260928 10:52:00.849015 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:00.837954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:52667 (local address 127.5.227.190:33507)
0928 10:52:00.838227 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:00.838233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:00.838468 (+ 235us) server_negotiation.cc:407] Connection header received
0928 10:52:00.838757 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:00.838765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:00.838845 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:00.838980 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:00.840066 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.841068 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.842591 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:00.843110 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:00.843261 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:00.848555 (+ 5294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:52667: BlockingRecv error: recv got EOF from 127.5.227.131:52667 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 28 10:52:01 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:01 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:01.069089 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:01.058186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:46133)
0928 10:52:01.058956 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:01.058974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:01.059163 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:01.059599 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:01.059604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:01.060202 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:01.060375 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:01.060384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.061478 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.061486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:01.062492 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:01.062504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.063273 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.063279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:01.063480 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:01.064143 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:01.064162 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:01.068610 (+ 4448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":400,"threads_started":1}
W20260928 10:52:01.069195 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:01.058374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:46133 (local address 127.5.227.190:33507)
0928 10:52:01.058753 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:01.058760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:01.059092 (+ 332us) server_negotiation.cc:407] Connection header received
0928 10:52:01.059356 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:01.059363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:01.059446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:01.059568 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:01.060546 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.061310 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.062667 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.063107 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.063252 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:01.068960 (+ 5708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:46133: BlockingRecv error: recv got EOF from 127.5.227.130:46133 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Sep 28 10:52:01 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:01 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:01.484519 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:01.474305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:51997)
0928 10:52:01.474926 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:01.474943 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:01.475143 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:01.475555 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:01.475560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:01.476071 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:01.476223 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:01.476232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.477332 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.477339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:01.478296 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:01.478307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.479085 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.479091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:01.479287 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:01.480323 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:01.480343 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:01.484157 (+ 3814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":342,"threads_started":1}
W20260928 10:52:01.484620 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:01.474484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:51997 (local address 127.5.227.190:33507)
0928 10:52:01.474746 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:01.474769 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:01.475056 (+ 287us) server_negotiation.cc:407] Connection header received
0928 10:52:01.475335 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:01.475345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:01.475413 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:01.475524 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:01.476382 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.477174 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.478454 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.478923 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.479069 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:01.484377 (+ 5308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:51997: BlockingRecv error: recv got EOF from 127.5.227.129:51997 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 10:52:01 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:01 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:01.860852 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:01.849652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:35247)
0928 10:52:01.850309 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:01.850327 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:01.850524 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:01.850936 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:01.850941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:01.851530 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:01.851710 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:01.851721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.852887 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.852894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:01.854336 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:01.854356 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.855296 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.855303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:01.855497 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:01.856261 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:01.856283 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:01.860593 (+ 4310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":364,"threads_started":1}
W20260928 10:52:01.861285 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:01.849786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:35247 (local address 127.5.227.190:33507)
0928 10:52:01.850097 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:01.850107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:01.850432 (+ 325us) server_negotiation.cc:407] Connection header received
0928 10:52:01.850693 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:01.850698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:01.850770 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:01.850901 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:01.851911 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.852729 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.854517 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:01.855065 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:01.855252 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:01.861034 (+ 5782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:35247: BlockingRecv error: recv got EOF from 127.5.227.131:35247 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 10:52:02 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:02 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:02.081557 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:02.070229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:49119)
0928 10:52:02.070810 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:02.070837 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:02.070962 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:02.071368 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:02.071372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:02.072034 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:02.072216 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:02.072274 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.073455 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.073460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:02.074766 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:02.074793 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.076060 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.076067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:02.076297 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:02.077120 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:02.077139 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:02.081294 (+ 4155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":331,"threads_started":1}
W20260928 10:52:02.081856 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:02.070412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:49119 (local address 127.5.227.190:33507)
0928 10:52:02.070599 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:02.070604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:02.070951 (+ 347us) server_negotiation.cc:407] Connection header received
0928 10:52:02.071086 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:02.071090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:02.071176 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:02.071326 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:02.072463 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.073288 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.074965 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.075734 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.075935 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:02.081556 (+ 5621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:49119: BlockingRecv error: recv got EOF from 127.5.227.130:49119 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 10:52:02 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:02 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:02.499167 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:02.486070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:50015)
0928 10:52:02.486850 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:02.486868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:02.487073 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:02.487504 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:02.487509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:02.488131 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:02.488427 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:02.488440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.490119 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.490126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:02.491660 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:02.491680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.492792 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.492799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:02.492999 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:02.493707 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:02.493952 (+ 245us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:02.498886 (+ 4934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":467,"threads_started":1}
W20260928 10:52:02.499553 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:02.486229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:50015 (local address 127.5.227.190:33507)
0928 10:52:02.486486 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:02.486496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:02.486978 (+ 482us) server_negotiation.cc:407] Connection header received
0928 10:52:02.487240 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:02.487249 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:02.487350 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:02.487466 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:02.488651 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.489937 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.492082 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.492577 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.492743 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:02.499252 (+ 6509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:50015: BlockingRecv error: recv got EOF from 127.5.227.129:50015 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 10:52:02 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:02 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.131@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:02.873775 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:02.862008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.131:40257)
0928 10:52:02.862497 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:02.862513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:02.862646 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:02.863044 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:02.863055 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:02.863567 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:02.863716 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:02.863942 (+ 226us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.865505 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.865510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:02.866881 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:02.866898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.867752 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.867758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:02.867953 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:02.868755 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:02.868779 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:02.873498 (+ 4719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":332,"threads_started":1}
W20260928 10:52:02.874365 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:02.862094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:40257 (local address 127.5.227.190:33507)
0928 10:52:02.862341 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:02.862348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:02.862618 (+ 270us) server_negotiation.cc:407] Connection header received
0928 10:52:02.862765 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:02.862770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:02.862864 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:02.862972 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:02.864161 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.865237 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.867053 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:02.867563 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:02.867674 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:02.874038 (+ 6364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.131:40257: BlockingRecv error: recv got EOF from 127.5.227.131:40257 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 28 10:52:03 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:03 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.130@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:03.095428 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:03.083089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.130:59067)
0928 10:52:03.083817 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:03.083835 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:03.083962 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:03.084453 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:03.084459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:03.085000 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:03.085208 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:03.085220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.086567 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.086572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:03.088300 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:03.088321 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.089360 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.089366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:03.089518 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:03.090491 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:03.090512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:03.095097 (+ 4585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":392,"threads_started":1}
W20260928 10:52:03.095816 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:03.083288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:59067 (local address 127.5.227.190:33507)
0928 10:52:03.083487 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:03.083492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:03.083927 (+ 435us) server_negotiation.cc:407] Connection header received
0928 10:52:03.084113 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:03.084118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:03.084252 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:03.084380 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:03.085393 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.086391 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.088473 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.089169 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.089296 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:03.095414 (+ 6118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.130:59067: BlockingRecv error: recv got EOF from 127.5.227.130:59067 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 10:52:03 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:52:03 dist-test-slave-8w8g krb5kdc[14284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.227.129@KRBTEST.COM for oryx/127.5.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:52:03.512244 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:03.500371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:33507 (local address 127.5.227.129:41441)
0928 10:52:03.500944 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:03.500962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:03.501096 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:03.501512 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:03.501516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:03.501957 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:03.502199 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:03.502210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.503341 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.503345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:03.504693 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:03.504940 (+ 247us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.506075 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.506080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:03.506267 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:03.507050 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:03.507075 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:03.511838 (+ 4763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.227.190:33507: Server oryx/127.5.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":352,"threads_started":1}
W20260928 10:52:03.512370 15022 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:03.500503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:41441 (local address 127.5.227.190:33507)
0928 10:52:03.500763 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:03.500767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:03.501082 (+ 315us) server_negotiation.cc:407] Connection header received
0928 10:52:03.501263 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:03.501267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:03.501329 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:03.501423 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:03.502389 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.503165 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.505118 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.505855 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.505988 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:03.512081 (+ 6093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.227.129:41441: BlockingRecv error: recv got EOF from 127.5.227.129:41441 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:52:03.545639 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 14873
I20260928 10:52:03.561299 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:03.547134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.5.227.129:35453)
0928 10:52:03.548061 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:03.548068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:03.548101 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:52:03.548271 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:03.548278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:03.548517 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:03.548706 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:03.550420 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.551783 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.553325 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.553978 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.554149 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:03.557222 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:03.557233 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:03.557251 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:03.557303 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:03.560518 (+ 3215us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:52:03.560938 (+ 420us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54186: Request ticket server kudu/127.5.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":471,"threads_started":1}
W20260928 10:52:03.561444 15336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54186: Request ticket server kudu/127.5.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:52:03.561460 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 14873: Not authorized: Client connection negotiation failed: client connection to 127.5.227.129:35453: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:52:03.561684 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 14873
I20260928 10:52:03.584869 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15024
I20260928 10:52:03.602715 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:03.586352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.5.227.130:35485)
0928 10:52:03.587303 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:03.587311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:03.587345 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:52:03.587418 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:03.587424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:03.587677 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:03.587936 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:03.590393 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.591632 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.593518 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.594526 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.594911 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:03.598157 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:03.598167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:03.598183 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:03.598235 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:03.601824 (+ 3589us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:52:03.602302 (+ 478us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59878: Request ticket server kudu/127.5.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":634,"threads_started":1}
W20260928 10:52:03.602835 15337 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59878: Request ticket server kudu/127.5.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:52:03.602964 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15024: Not authorized: Client connection negotiation failed: client connection to 127.5.227.130:35485: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:52:03.603166 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15024
I20260928 10:52:03.625540 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15157
I20260928 10:52:03.641112 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:03.626886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.5.227.131:38455)
0928 10:52:03.627648 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:03.627659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:03.627697 (+ 38us) server_negotiation.cc:407] Connection header received
0928 10:52:03.627796 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:03.627806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:03.628106 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:03.628424 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:03.629855 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.630899 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.632452 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:03.633112 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:03.633311 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:03.636677 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:03.636687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:03.636711 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:03.636772 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:03.640133 (+ 3361us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:52:03.640608 (+ 475us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43676: Request ticket server kudu/127.5.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":295,"threads_started":1}
W20260928 10:52:03.641216 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15157: Not authorized: Client connection negotiation failed: client connection to 127.5.227.131:38455: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:52:03.641312 15338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43676: Request ticket server kudu/127.5.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:52:03.641453 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15157
I20260928 10:52:03.660550 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 14804
I20260928 10:52:03.662009 14866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:03.784847 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 14804
2026-09-28T10:52:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22549 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T10:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:52:03Z Disabled control of system clock
I20260928 10:52:03.874542 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37677
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 10:52:04.254091 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:04.254456 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:04.254544 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:04.263137 15343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:52:04.263288 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:04.263327 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:04.263378 15343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:52:04.263414 15343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:52:04.273192 15343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:37677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:04.274569 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:04.276408 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:04.290050 15349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:04.290014 15348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:04.290496 15343 server_base.cc:1034] running on GCE node
W20260928 10:52:04.290078 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:04.292038 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:04.293736 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:04.295025 15343 hybrid_clock.cc:648] HybridClock initialized: now 1790592724294972 us; error 38 us; skew 500 ppm
I20260928 10:52:04.295548 15343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:04.302678 15343 webserver.cc:460] Webserver started at http://127.5.227.190:36119/ using document root <none> and password file <none>
I20260928 10:52:04.303593 15343 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:04.303781 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:04.304168 15343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:04.307602 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "f7830b31730e454eb6a8da84ec97a482"
format_stamp: "Formatted at 2026-09-28 10:52:04 on dist-test-slave-8w8g"
server_key: "eff09b300bf4187135d68c2f69647e7b"
server_key_iv: "9f10496490b903f9a225fd0e8a9682fc"
server_key_version: "encryptionkey@0"
I20260928 10:52:04.308681 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "f7830b31730e454eb6a8da84ec97a482"
format_stamp: "Formatted at 2026-09-28 10:52:04 on dist-test-slave-8w8g"
server_key: "eff09b300bf4187135d68c2f69647e7b"
server_key_iv: "9f10496490b903f9a225fd0e8a9682fc"
server_key_version: "encryptionkey@0"
I20260928 10:52:04.315193 15343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:52:04.319941 15357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:04.322010 15343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:52:04.322279 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "f7830b31730e454eb6a8da84ec97a482"
format_stamp: "Formatted at 2026-09-28 10:52:04 on dist-test-slave-8w8g"
server_key: "eff09b300bf4187135d68c2f69647e7b"
server_key_iv: "9f10496490b903f9a225fd0e8a9682fc"
server_key_version: "encryptionkey@0"
I20260928 10:52:04.322563 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:04.343571 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:04.344781 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:04.345145 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:04.367554 15343 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:37677
I20260928 10:52:04.367590 15408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:37677 every 8 connection(s)
I20260928 10:52:04.369112 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:52:04.372473 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15343
I20260928 10:52:04.372710 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:52:04.373495 6030 external_mini_cluster.cc:1428] Setting key c5dab11a21de325b1ffca605434e5451
I20260928 10:52:04.374027 15409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:52:04.391448 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482: Bootstrap starting.
I20260928 10:52:04.391629 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:04.376017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.5.227.190:37677)
0928 10:52:04.377221 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:04.377233 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:04.377288 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:52:04.379831 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:04.379861 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:04.380426 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:04.380814 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:04.381655 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:04.383123 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:04.385507 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:04.386299 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:04.386460 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:04.388073 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:04.388082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:04.388100 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:04.388900 (+ 800us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:04.389078 (+ 178us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:04.389119 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:04.389235 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:04.389575 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 10:52:04.390065 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":446,"threads_started":1}
I20260928 10:52:04.396489 15409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482: Neither blocks nor log segments found. Creating new log.
I20260928 10:52:04.397918 15409 log.cc:824] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482: Log is configured to *not* fsync() on all Append() calls
I20260928 10:52:04.401636 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482: No bootstrap required, opened a new log
I20260928 10:52:04.411571 15409 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7830b31730e454eb6a8da84ec97a482" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37677 } }
I20260928 10:52:04.412099 15409 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:52:04.412194 15409 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7830b31730e454eb6a8da84ec97a482, State: Initialized, Role: FOLLOWER
I20260928 10:52:04.412811 15409 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7830b31730e454eb6a8da84ec97a482" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37677 } }
I20260928 10:52:04.413035 15409 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:52:04.413131 15409 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:52:04.413275 15409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:52:04.416230 15409 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7830b31730e454eb6a8da84ec97a482" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37677 } }
I20260928 10:52:04.417075 15409 leader_election.cc:302] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7830b31730e454eb6a8da84ec97a482; no voters:
I20260928 10:52:04.417879 15409 leader_election.cc:288] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:52:04.418308 15414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:52:04.419036 15414 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [term 1 LEADER]: Becoming Leader. State: Replica: f7830b31730e454eb6a8da84ec97a482, State: Running, Role: LEADER
I20260928 10:52:04.419611 15414 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7830b31730e454eb6a8da84ec97a482" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37677 } }
I20260928 10:52:04.420615 15409 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:52:04.423199 15415 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7830b31730e454eb6a8da84ec97a482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7830b31730e454eb6a8da84ec97a482" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37677 } } }
I20260928 10:52:04.423208 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7830b31730e454eb6a8da84ec97a482. Latest consensus state: current_term: 1 leader_uuid: "f7830b31730e454eb6a8da84ec97a482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7830b31730e454eb6a8da84ec97a482" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37677 } } }
I20260928 10:52:04.423918 15415 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [sys.catalog]: This master's current role is: LEADER
I20260928 10:52:04.423991 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482 [sys.catalog]: This master's current role is: LEADER
I20260928 10:52:04.429431 15422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:52:04.436064 15422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:52:04.448246 15422 catalog_manager.cc:1347] Generated new cluster ID: 825b4d72506a4e459f8c94005e983b2d
I20260928 10:52:04.448406 15422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:52:04.458246 15422 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:52:04.460012 15422 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:52:04.479650 15422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7830b31730e454eb6a8da84ec97a482: Generated new TSK 0
I20260928 10:52:04.480741 15422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:52:04.494630 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37677
--builtin_ntp_servers=127.5.227.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 10:52:04.884541 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:04.884974 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:04.885200 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:04.893638 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:04.893770 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:04.893880 15433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:52:04.904875 15433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:04.906280 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:04.908229 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:04.923830 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:04.923943 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:04.923767 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:04.925559 15433 server_base.cc:1034] running on GCE node
I20260928 10:52:04.926422 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:04.928033 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:04.929327 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790592724929299 us; error 68 us; skew 500 ppm
I20260928 10:52:04.929762 15433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:04.932771 15433 webserver.cc:460] Webserver started at http://127.5.227.129:46461/ using document root <none> and password file <none>
I20260928 10:52:04.933588 15433 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:04.933697 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:04.934039 15433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:04.937459 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "d5a64a80cc9547acbb27d44549b48ee7"
format_stamp: "Formatted at 2026-09-28 10:52:04 on dist-test-slave-8w8g"
server_key: "4fd7be2cb9a6eebe97d896362edb9596"
server_key_iv: "ec9b6e082c7aee4d78a46c8e67d960fb"
server_key_version: "encryptionkey@0"
I20260928 10:52:04.938668 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "d5a64a80cc9547acbb27d44549b48ee7"
format_stamp: "Formatted at 2026-09-28 10:52:04 on dist-test-slave-8w8g"
server_key: "4fd7be2cb9a6eebe97d896362edb9596"
server_key_iv: "ec9b6e082c7aee4d78a46c8e67d960fb"
server_key_version: "encryptionkey@0"
I20260928 10:52:04.946341 15433 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 10:52:04.950997 15447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:04.952934 15433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:52:04.953181 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "d5a64a80cc9547acbb27d44549b48ee7"
format_stamp: "Formatted at 2026-09-28 10:52:04 on dist-test-slave-8w8g"
server_key: "4fd7be2cb9a6eebe97d896362edb9596"
server_key_iv: "ec9b6e082c7aee4d78a46c8e67d960fb"
server_key_version: "encryptionkey@0"
I20260928 10:52:04.953446 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:04.970759 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:04.971886 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:04.972256 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:04.973376 15433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:04.975078 15433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:04.975243 15433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:04.975440 15433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:04.975548 15433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:05.017228 15433 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:38243
I20260928 10:52:05.017275 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:38243 every 8 connection(s)
I20260928 10:52:05.018707 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:52:05.022257 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15433
I20260928 10:52:05.022513 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:52:05.023370 6030 external_mini_cluster.cc:1428] Setting key 65fd9406938cc494bdf2bc1c04f1bfbc
I20260928 10:52:05.026783 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37677
--builtin_ntp_servers=127.5.227.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 10:52:05.035180 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:05.021678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:60913 (local address 127.5.227.190:37677)
0928 10:52:05.022266 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:05.022273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:05.023985 (+ 1712us) server_negotiation.cc:407] Connection header received
0928 10:52:05.026261 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:05.026268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:05.026360 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:05.026493 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:05.027815 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.029211 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.031128 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.031979 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.032139 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:05.033987 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:05.033995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:05.034002 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:05.034035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:05.034094 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:05.034117 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:05.034229 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:05.034578 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 10:52:05.034715 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":310,"threads_started":1}
I20260928 10:52:05.036356 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:05.022378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37677 (local address 127.5.227.129:60913)
0928 10:52:05.023785 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:05.023834 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:05.025974 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:05.026763 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:05.026781 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:05.026897 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:05.027583 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:05.027600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.029397 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.029402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:05.030950 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:05.030964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.032317 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.032323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:05.032557 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:05.033549 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:52:05.033552 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:05.033601 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:52:05.033712 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:05.034314 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:05.034324 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:05.034330 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:52:05.034533 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 10:52:05.034902 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":565,"threads_started":1}
I20260928 10:52:05.038558 15560 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37677
I20260928 10:52:05.038920 15560 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:05.039995 15560 heartbeater.cc:511] Master 127.5.227.190:37677 requested a full tablet report, sending...
I20260928 10:52:05.042613 15374 ts_manager.cc:194] Registered new tserver with Master: d5a64a80cc9547acbb27d44549b48ee7 (127.5.227.129:38243)
I20260928 10:52:05.044427 15374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.129:60913
W20260928 10:52:05.396054 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:05.396466 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:05.396622 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:05.404743 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:05.404917 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:05.405102 15564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:52:05.416021 15564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:05.417484 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:05.419334 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:05.435652 15569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:05.435696 15570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:05.435838 15572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:05.436638 15564 server_base.cc:1034] running on GCE node
I20260928 10:52:05.437470 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:05.438609 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:05.442004 15564 hybrid_clock.cc:648] HybridClock initialized: now 1790592725441820 us; error 163 us; skew 500 ppm
I20260928 10:52:05.442430 15564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:05.445611 15564 webserver.cc:460] Webserver started at http://127.5.227.130:43525/ using document root <none> and password file <none>
I20260928 10:52:05.446532 15564 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:05.446662 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:05.447089 15564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:05.450989 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "bc8b39cad3c84f62bd5d55cd3b799574"
format_stamp: "Formatted at 2026-09-28 10:52:05 on dist-test-slave-8w8g"
server_key: "ca3bb3cf276b60173ccb2dff69056bcd"
server_key_iv: "f5db0efb97ecffe56b6e117e5304b78b"
server_key_version: "encryptionkey@0"
I20260928 10:52:05.452104 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "bc8b39cad3c84f62bd5d55cd3b799574"
format_stamp: "Formatted at 2026-09-28 10:52:05 on dist-test-slave-8w8g"
server_key: "ca3bb3cf276b60173ccb2dff69056bcd"
server_key_iv: "f5db0efb97ecffe56b6e117e5304b78b"
server_key_version: "encryptionkey@0"
I20260928 10:52:05.458902 15564 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:52:05.463168 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:05.464991 15564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:52:05.465227 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "bc8b39cad3c84f62bd5d55cd3b799574"
format_stamp: "Formatted at 2026-09-28 10:52:05 on dist-test-slave-8w8g"
server_key: "ca3bb3cf276b60173ccb2dff69056bcd"
server_key_iv: "f5db0efb97ecffe56b6e117e5304b78b"
server_key_version: "encryptionkey@0"
I20260928 10:52:05.465510 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:05.487002 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:05.488111 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:05.488477 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:05.489563 15564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:05.491279 15564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:05.491420 15564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:05.491566 15564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:05.491671 15564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:05.530593 15564 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:42031
I20260928 10:52:05.530689 15690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:42031 every 8 connection(s)
I20260928 10:52:05.532171 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:52:05.537928 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15564
I20260928 10:52:05.538136 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:52:05.538676 6030 external_mini_cluster.cc:1428] Setting key e01199e50d414a3d16e107d5432f41e7
I20260928 10:52:05.541867 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37677
--builtin_ntp_servers=127.5.227.148:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 10:52:05.548704 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:05.535109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:47485 (local address 127.5.227.190:37677)
0928 10:52:05.535386 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:05.535391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:05.537081 (+ 1690us) server_negotiation.cc:407] Connection header received
0928 10:52:05.539633 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:05.539638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:05.539766 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:05.539906 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:05.541182 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.542150 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.545242 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.545739 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.545918 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:05.547864 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:05.547870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:05.547874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:05.547897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:05.547934 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:05.547949 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:05.548046 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:05.548360 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 10:52:05.548450 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260928 10:52:05.550153 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:05.535653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37677 (local address 127.5.227.130:47485)
0928 10:52:05.536868 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:05.536912 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:05.539317 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:05.540169 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:05.540178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:05.540286 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:05.540929 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:05.540954 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.542331 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.542339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:05.544965 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:05.544992 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:05.546067 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:05.546076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:05.546445 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:05.547343 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:52:05.547350 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:05.547414 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:52:05.547532 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:05.548077 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:05.548091 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:05.548102 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:52:05.548314 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 10:52:05.548781 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":358,"threads_started":1}
I20260928 10:52:05.552237 15691 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37677
I20260928 10:52:05.552582 15691 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:05.553423 15691 heartbeater.cc:511] Master 127.5.227.190:37677 requested a full tablet report, sending...
I20260928 10:52:05.555186 15374 ts_manager.cc:194] Registered new tserver with Master: bc8b39cad3c84f62bd5d55cd3b799574 (127.5.227.130:42031)
I20260928 10:52:05.556375 15374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.130:47485
W20260928 10:52:05.925997 15694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:05.926363 15694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:05.926520 15694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:05.935314 15694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:05.935504 15694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:05.935693 15694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:52:05.947388 15694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:05.948889 15694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:05.950815 15694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:05.968110 15702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:05.968283 15699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:05.968137 15700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:05.970633 15694 server_base.cc:1034] running on GCE node
I20260928 10:52:05.971387 15694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:05.972566 15694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:05.973930 15694 hybrid_clock.cc:648] HybridClock initialized: now 1790592725973876 us; error 38 us; skew 500 ppm
I20260928 10:52:05.974346 15694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:05.977501 15694 webserver.cc:460] Webserver started at http://127.5.227.131:45939/ using document root <none> and password file <none>
I20260928 10:52:05.978302 15694 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:05.978404 15694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:05.978817 15694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:05.982412 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "bb9c90ddc80745dab2a6a5d8344aa55a"
format_stamp: "Formatted at 2026-09-28 10:52:05 on dist-test-slave-8w8g"
server_key: "d5d5ed776ea37b189a5c97d1df38d1f7"
server_key_iv: "b8777eef12ab996893f2cc5591160564"
server_key_version: "encryptionkey@0"
I20260928 10:52:05.983386 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "bb9c90ddc80745dab2a6a5d8344aa55a"
format_stamp: "Formatted at 2026-09-28 10:52:05 on dist-test-slave-8w8g"
server_key: "d5d5ed776ea37b189a5c97d1df38d1f7"
server_key_iv: "b8777eef12ab996893f2cc5591160564"
server_key_version: "encryptionkey@0"
I20260928 10:52:05.990855 15694 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:52:05.995190 15708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:05.997113 15694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 10:52:05.997377 15694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "bb9c90ddc80745dab2a6a5d8344aa55a"
format_stamp: "Formatted at 2026-09-28 10:52:05 on dist-test-slave-8w8g"
server_key: "d5d5ed776ea37b189a5c97d1df38d1f7"
server_key_iv: "b8777eef12ab996893f2cc5591160564"
server_key_version: "encryptionkey@0"
I20260928 10:52:05.997639 15694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:06.017902 15694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:06.019086 15694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:06.019838 15694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:06.022113 15694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:06.025425 15694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:06.025604 15694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:06.025714 15694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:06.025782 15694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:06.048317 15560 heartbeater.cc:503] Master 127.5.227.190:37677 was elected leader, sending a full tablet report...
I20260928 10:52:06.071277 15694 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:41303
I20260928 10:52:06.071548 15820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:41303 every 8 connection(s)
I20260928 10:52:06.073438 15694 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:52:06.083845 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15694
I20260928 10:52:06.084038 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:52:06.084546 6030 external_mini_cluster.cc:1428] Setting key ffffc75d44895132b076bdfbf512fbdd
I20260928 10:52:06.089465 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:06.076833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:43047 (local address 127.5.227.190:37677)
0928 10:52:06.077330 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:06.077335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:06.078766 (+ 1431us) server_negotiation.cc:407] Connection header received
0928 10:52:06.080747 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:06.080753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:06.080857 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:06.080998 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:06.082548 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.083347 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.085834 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.086420 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.086567 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:06.088324 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:06.088330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:06.088340 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:06.088365 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:06.088421 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:06.088434 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:06.088562 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:06.089007 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 10:52:06.089101 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":269,"threads_started":1}
I20260928 10:52:06.090497 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:06.077447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37677 (local address 127.5.227.131:43047)
0928 10:52:06.078545 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:06.078599 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:06.080403 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:06.081394 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:06.081405 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:06.081511 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:06.082325 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:06.082345 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.083525 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.083532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:06.085558 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:06.085576 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.086666 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.086673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:06.086939 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:06.087957 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:52:06.087960 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:06.088012 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:52:06.088105 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:06.088600 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:06.088609 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:06.088616 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:52:06.088898 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 10:52:06.089299 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":403,"threads_started":1}
I20260928 10:52:06.092346 15821 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37677
I20260928 10:52:06.092705 15821 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:06.093575 15821 heartbeater.cc:511] Master 127.5.227.190:37677 requested a full tablet report, sending...
I20260928 10:52:06.095372 15374 ts_manager.cc:194] Registered new tserver with Master: bb9c90ddc80745dab2a6a5d8344aa55a (127.5.227.131:41303)
I20260928 10:52:06.096288 15374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.131:43047
I20260928 10:52:06.102962 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:52:06.111368 15823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:06.109435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.5.227.190:37677)
0928 10:52:06.109882 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:06.109887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:06.110202 (+ 315us) server_negotiation.cc:407] Connection header received
0928 10:52:06.110325 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:06.110330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:06.110400 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:06.110518 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:06.110966 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41494: BlockingRecv error: recv got EOF from 127.0.0.1:41494 (error 108)
Metrics: {"server-negotiator.queue_time_us":318}
I20260928 10:52:06.113313 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15433
I20260928 10:52:06.124914 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:06.114953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.5.227.129:38243)
0928 10:52:06.115889 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:06.115897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:06.115926 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:52:06.115987 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:06.115994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:06.116237 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:06.116384 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:06.117130 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.118235 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.120283 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.121070 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.121214 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:06.122900 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:06.122920 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:06.122937 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:06.123634 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:06.123702 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:06.123806 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:06.123924 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:06.124221 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 10:52:06.124363 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":543,"threads_started":1}
I20260928 10:52:06.126492 15555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:06.229481 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15433
I20260928 10:52:06.249428 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15564
I20260928 10:52:06.261228 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:06.251092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.5.227.130:42031)
0928 10:52:06.252006 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:06.252014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:06.252045 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:52:06.252119 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:06.252125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:06.252359 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:06.252504 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:06.253584 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.255056 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.256735 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.257487 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.257616 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:06.258952 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:06.258961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:06.258977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:06.259723 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:06.259871 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:06.259897 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:06.260002 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:06.260477 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 10:52:06.260892 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":320,"threads_started":1}
I20260928 10:52:06.262712 15686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:06.357368 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15564
I20260928 10:52:06.381410 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15694
I20260928 10:52:06.391237 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:06.382562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.5.227.131:41303)
0928 10:52:06.383313 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:06.383321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:06.383355 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:52:06.383419 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:06.383425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:06.383661 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:06.384006 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:06.384476 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.385539 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.386853 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:06.387532 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:06.387688 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:06.389267 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:06.389284 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:06.389300 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:06.390109 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:06.390162 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:06.390197 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:06.390348 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:06.390508 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 10:52:06.390669 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":415,"threads_started":1}
I20260928 10:52:06.392848 15816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:06.486007 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15694
I20260928 10:52:06.504503 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15343
I20260928 10:52:06.506006 15404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:06.619432 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15343
2026-09-28T10:52:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2824 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T10:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:52:06Z Disabled control of system clock
I20260928 10:52:06.700364 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:34695
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:34695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 10:52:07.082508 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:07.082845 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:07.082933 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:07.091220 15841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:52:07.091365 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:07.091404 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:07.091456 15841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:52:07.091495 15841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:52:07.102229 15841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:34695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:34695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:07.103622 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:07.105489 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:07.120088 15846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:07.120143 15847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:07.120200 15849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:07.126220 15841 server_base.cc:1034] running on GCE node
I20260928 10:52:07.127218 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:07.128983 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:07.130259 15841 hybrid_clock.cc:648] HybridClock initialized: now 1790592727130216 us; error 42 us; skew 500 ppm
I20260928 10:52:07.130712 15841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:07.133594 15841 webserver.cc:460] Webserver started at http://127.5.227.190:37361/ using document root <none> and password file <none>
I20260928 10:52:07.134486 15841 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:07.134644 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:07.135025 15841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:07.138712 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "b8fb3d0b00fa40038d36cdc44a36e404"
format_stamp: "Formatted at 2026-09-28 10:52:07 on dist-test-slave-8w8g"
server_key: "b68c5a7378f4f1e132e91b2e32611814"
server_key_iv: "42b30d2c3346bb3d7ff74aa87df4ab51"
server_key_version: "encryptionkey@0"
I20260928 10:52:07.140348 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "b8fb3d0b00fa40038d36cdc44a36e404"
format_stamp: "Formatted at 2026-09-28 10:52:07 on dist-test-slave-8w8g"
server_key: "b68c5a7378f4f1e132e91b2e32611814"
server_key_iv: "42b30d2c3346bb3d7ff74aa87df4ab51"
server_key_version: "encryptionkey@0"
I20260928 10:52:07.146687 15841 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:52:07.152026 15855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:07.153973 15841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:52:07.154443 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "b8fb3d0b00fa40038d36cdc44a36e404"
format_stamp: "Formatted at 2026-09-28 10:52:07 on dist-test-slave-8w8g"
server_key: "b68c5a7378f4f1e132e91b2e32611814"
server_key_iv: "42b30d2c3346bb3d7ff74aa87df4ab51"
server_key_version: "encryptionkey@0"
I20260928 10:52:07.154747 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:07.162488 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:07.163452 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:07.163818 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:07.187994 15841 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:34695
I20260928 10:52:07.188045 15906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:34695 every 8 connection(s)
I20260928 10:52:07.189551 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:52:07.189906 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15841
I20260928 10:52:07.190105 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:52:07.191063 6030 external_mini_cluster.cc:1428] Setting key 9ca6705952dedbcb18c33104184b323e
I20260928 10:52:07.194984 15907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:52:07.201205 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:07.193386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.5.227.190:34695)
0928 10:52:07.194523 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:07.194535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:07.194580 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:52:07.196628 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:07.196652 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:07.197139 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:07.197603 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:07.199152 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:07.199171 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:07.199187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:07.199204 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:07.199301 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:07.199320 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:07.199596 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:07.199805 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 10:52:07.200097 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":414,"threads_started":1}
I20260928 10:52:07.208828 15907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404: Bootstrap starting.
I20260928 10:52:07.213410 15907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404: Neither blocks nor log segments found. Creating new log.
I20260928 10:52:07.214692 15907 log.cc:824] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404: Log is configured to *not* fsync() on all Append() calls
I20260928 10:52:07.218691 15907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404: No bootstrap required, opened a new log
I20260928 10:52:07.226872 15907 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34695 } }
I20260928 10:52:07.227279 15907 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:52:07.227423 15907 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8fb3d0b00fa40038d36cdc44a36e404, State: Initialized, Role: FOLLOWER
I20260928 10:52:07.228132 15907 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34695 } }
I20260928 10:52:07.228343 15907 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:52:07.228510 15907 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:52:07.228704 15907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:52:07.231457 15907 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34695 } }
I20260928 10:52:07.232077 15907 leader_election.cc:302] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8fb3d0b00fa40038d36cdc44a36e404; no voters:
I20260928 10:52:07.232801 15907 leader_election.cc:288] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:52:07.233009 15912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:52:07.234445 15912 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [term 1 LEADER]: Becoming Leader. State: Replica: b8fb3d0b00fa40038d36cdc44a36e404, State: Running, Role: LEADER
I20260928 10:52:07.235040 15912 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34695 } }
I20260928 10:52:07.235512 15907 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:52:07.238251 15914 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8fb3d0b00fa40038d36cdc44a36e404. Latest consensus state: current_term: 1 leader_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34695 } } }
I20260928 10:52:07.238682 15914 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [sys.catalog]: This master's current role is: LEADER
I20260928 10:52:07.239169 15913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fb3d0b00fa40038d36cdc44a36e404" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34695 } } }
I20260928 10:52:07.239667 15913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404 [sys.catalog]: This master's current role is: LEADER
I20260928 10:52:07.242874 15921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:52:07.249272 15921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:52:07.261477 15921 catalog_manager.cc:1347] Generated new cluster ID: dca4543bcae2412c9e29190339e05077
I20260928 10:52:07.261610 15921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:52:07.288277 15921 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:52:07.288913 15921 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:52:07.298388 15921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8fb3d0b00fa40038d36cdc44a36e404: Generated new TSK 0
I20260928 10:52:07.299404 15921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:52:07.307451 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:34695
--builtin_ntp_servers=127.5.227.148:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 10:52:07.741088 15931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:07.741401 15931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:07.741561 15931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:07.750391 15931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:07.750568 15931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:07.750910 15931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:52:07.764286 15931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:34695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:07.766199 15931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:07.768464 15931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:07.787403 15937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:07.787456 15939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:07.788079 15936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:07.788816 15931 server_base.cc:1034] running on GCE node
I20260928 10:52:07.789630 15931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:07.790796 15931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:07.792268 15931 hybrid_clock.cc:648] HybridClock initialized: now 1790592727792320 us; error 154 us; skew 500 ppm
I20260928 10:52:07.792778 15931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:07.796039 15931 webserver.cc:460] Webserver started at http://127.5.227.129:36223/ using document root <none> and password file <none>
I20260928 10:52:07.796947 15931 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:07.797109 15931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:07.797497 15931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:07.801241 15931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "d988de0cfe9a4dda837ca2d62f66fffd"
format_stamp: "Formatted at 2026-09-28 10:52:07 on dist-test-slave-8w8g"
server_key: "4bff00d21e1cca0d0df3695ae8ed4e3b"
server_key_iv: "f26cb1618e63929599689cb4b96146df"
server_key_version: "encryptionkey@0"
I20260928 10:52:07.802240 15931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "d988de0cfe9a4dda837ca2d62f66fffd"
format_stamp: "Formatted at 2026-09-28 10:52:07 on dist-test-slave-8w8g"
server_key: "4bff00d21e1cca0d0df3695ae8ed4e3b"
server_key_iv: "f26cb1618e63929599689cb4b96146df"
server_key_version: "encryptionkey@0"
I20260928 10:52:07.808378 15931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 10:52:07.813460 15945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:07.815848 15931 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.002s
I20260928 10:52:07.816073 15931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "d988de0cfe9a4dda837ca2d62f66fffd"
format_stamp: "Formatted at 2026-09-28 10:52:07 on dist-test-slave-8w8g"
server_key: "4bff00d21e1cca0d0df3695ae8ed4e3b"
server_key_iv: "f26cb1618e63929599689cb4b96146df"
server_key_version: "encryptionkey@0"
I20260928 10:52:07.816390 15931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:07.823071 15931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:07.824087 15931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:07.824574 15931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:07.826056 15931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:07.827911 15931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:07.828044 15931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:07.828176 15931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:07.828269 15931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:07.874534 15931 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:45075
I20260928 10:52:07.874586 16057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:45075 every 8 connection(s)
I20260928 10:52:07.876945 15931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:52:07.882218 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 15931
I20260928 10:52:07.882421 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:52:07.883091 6030 external_mini_cluster.cc:1428] Setting key 61d52af83436e02727d94370c2c76411
I20260928 10:52:07.885918 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:34695
--builtin_ntp_servers=127.5.227.148:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 10:52:07.887581 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:07.880319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:60815 (local address 127.5.227.190:34695)
0928 10:52:07.880902 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:07.880907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:07.882544 (+ 1637us) server_negotiation.cc:407] Connection header received
0928 10:52:07.884582 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:07.884589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:07.884660 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:07.884973 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:07.886652 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:07.886658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:07.886664 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:07.886693 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:07.886742 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:07.886756 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:07.886907 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:07.887130 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 10:52:07.887206 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":234,"threads_started":1}
I20260928 10:52:07.888722 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:07.881088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:34695 (local address 127.5.227.129:60815)
0928 10:52:07.882173 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:07.882219 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:07.884264 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:07.885248 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:07.885258 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:07.885368 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:07.886292 (+ 924us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:52:07.886295 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:07.886346 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:52:07.886448 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:07.886866 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:07.886878 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:07.886889 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:52:07.887230 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 10:52:07.887549 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":310,"threads_started":1}
I20260928 10:52:07.890985 16058 heartbeater.cc:347] Connected to a master server at 127.5.227.190:34695
I20260928 10:52:07.891455 16058 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:07.892272 16058 heartbeater.cc:511] Master 127.5.227.190:34695 requested a full tablet report, sending...
I20260928 10:52:07.895155 15872 ts_manager.cc:194] Registered new tserver with Master: d988de0cfe9a4dda837ca2d62f66fffd (127.5.227.129:45075)
W20260928 10:52:08.295099 16062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:08.295529 16062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:08.295712 16062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:08.305622 16062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:08.305796 16062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:08.305986 16062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:52:08.318013 16062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:34695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:08.319533 16062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:08.321959 16062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:08.341079 16068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:08.341876 16067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:08.342536 16070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:08.343417 16062 server_base.cc:1034] running on GCE node
I20260928 10:52:08.344313 16062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:08.345359 16062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:08.346927 16062 hybrid_clock.cc:648] HybridClock initialized: now 1790592728346871 us; error 39 us; skew 500 ppm
I20260928 10:52:08.347515 16062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:08.350630 16062 webserver.cc:460] Webserver started at http://127.5.227.130:43725/ using document root <none> and password file <none>
I20260928 10:52:08.351621 16062 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:08.351796 16062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:08.352177 16062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:08.355428 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "e3bfd9df50254858883bc98342abb842"
format_stamp: "Formatted at 2026-09-28 10:52:08 on dist-test-slave-8w8g"
server_key: "378303f959f33865ff57baac8280944a"
server_key_iv: "4d6a9c2eb0e83fefd7f01a4ae6ac0f5c"
server_key_version: "encryptionkey@0"
I20260928 10:52:08.356612 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "e3bfd9df50254858883bc98342abb842"
format_stamp: "Formatted at 2026-09-28 10:52:08 on dist-test-slave-8w8g"
server_key: "378303f959f33865ff57baac8280944a"
server_key_iv: "4d6a9c2eb0e83fefd7f01a4ae6ac0f5c"
server_key_version: "encryptionkey@0"
I20260928 10:52:08.365334 16062 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260928 10:52:08.370549 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:08.373162 16062 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260928 10:52:08.373420 16062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "e3bfd9df50254858883bc98342abb842"
format_stamp: "Formatted at 2026-09-28 10:52:08 on dist-test-slave-8w8g"
server_key: "378303f959f33865ff57baac8280944a"
server_key_iv: "4d6a9c2eb0e83fefd7f01a4ae6ac0f5c"
server_key_version: "encryptionkey@0"
I20260928 10:52:08.373670 16062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:08.381352 16062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:08.382863 16062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:08.383255 16062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:08.384770 16062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:08.386530 16062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:08.386623 16062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:08.386744 16062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:08.386819 16062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:08.429586 16062 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:41299
I20260928 10:52:08.429781 16188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:41299 every 8 connection(s)
I20260928 10:52:08.431020 16062 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:52:08.438825 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 16062
I20260928 10:52:08.439307 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:52:08.440160 6030 external_mini_cluster.cc:1428] Setting key 1da929d373d9124fd57d9086a8aabe60
I20260928 10:52:08.443348 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:08.435080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:60245 (local address 127.5.227.190:34695)
0928 10:52:08.435918 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:08.435924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:08.437103 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 10:52:08.439938 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:08.439942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:08.440019 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:08.440125 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:08.442108 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:08.442112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:08.442115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:08.442133 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:08.442179 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:08.442192 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:08.442366 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:08.442934 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 10:52:08.443010 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":385,"threads_started":1}
I20260928 10:52:08.444162 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:34695
--builtin_ntp_servers=127.5.227.148:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 10:52:08.444643 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:08.435636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:34695 (local address 127.5.227.130:60245)
0928 10:52:08.436935 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:08.436974 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:08.439461 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:08.440488 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:08.440502 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:08.440651 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:08.441546 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:52:08.441550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:08.441609 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:52:08.441746 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:08.442409 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:08.442431 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:08.442441 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 10:52:08.442927 (+ 486us) client_negotiation.cc:240] Negotiation successful
0928 10:52:08.443295 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":437,"threads_started":1}
I20260928 10:52:08.446252 16189 heartbeater.cc:347] Connected to a master server at 127.5.227.190:34695
I20260928 10:52:08.446604 16189 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:08.447561 16189 heartbeater.cc:511] Master 127.5.227.190:34695 requested a full tablet report, sending...
I20260928 10:52:08.450423 15872 ts_manager.cc:194] Registered new tserver with Master: e3bfd9df50254858883bc98342abb842 (127.5.227.130:41299)
W20260928 10:52:08.831321 16193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:08.831686 16193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:08.831887 16193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:08.840703 16193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:08.840835 16193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:08.840960 16193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:52:08.851150 16193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:34695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:08.852483 16193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:08.855363 16193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:08.871641 16201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:08.871762 16199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:08.871682 16198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:08.873250 16193 server_base.cc:1034] running on GCE node
I20260928 10:52:08.874176 16193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:08.876149 16193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:08.877463 16193 hybrid_clock.cc:648] HybridClock initialized: now 1790592728877384 us; error 69 us; skew 500 ppm
I20260928 10:52:08.877883 16193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:52:08.881414 16193 webserver.cc:460] Webserver started at http://127.5.227.131:36221/ using document root <none> and password file <none>
I20260928 10:52:08.882175 16193 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:08.882277 16193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:08.882552 16193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:08.885607 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "818b0985446840ea91bb73910caf1aaa"
format_stamp: "Formatted at 2026-09-28 10:52:08 on dist-test-slave-8w8g"
server_key: "20dc55fcb4b6b2dde1286f62d4314ac0"
server_key_iv: "5bd69eed1b77f52bc3e53988fe8e4faa"
server_key_version: "encryptionkey@0"
I20260928 10:52:08.886412 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "818b0985446840ea91bb73910caf1aaa"
format_stamp: "Formatted at 2026-09-28 10:52:08 on dist-test-slave-8w8g"
server_key: "20dc55fcb4b6b2dde1286f62d4314ac0"
server_key_iv: "5bd69eed1b77f52bc3e53988fe8e4faa"
server_key_version: "encryptionkey@0"
I20260928 10:52:08.892194 16193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 10:52:08.896521 16207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:08.898564 16193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:52:08.898890 16193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "818b0985446840ea91bb73910caf1aaa"
format_stamp: "Formatted at 2026-09-28 10:52:08 on dist-test-slave-8w8g"
server_key: "20dc55fcb4b6b2dde1286f62d4314ac0"
server_key_iv: "5bd69eed1b77f52bc3e53988fe8e4faa"
server_key_version: "encryptionkey@0"
I20260928 10:52:08.899219 16058 heartbeater.cc:503] Master 127.5.227.190:34695 was elected leader, sending a full tablet report...
I20260928 10:52:08.899178 16193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:08.906447 16193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:08.907356 16193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:08.907662 16193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:08.908891 16193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:08.910530 16193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:08.910624 16193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:08.910902 16193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:08.911020 16193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:08.954404 16193 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:34141
I20260928 10:52:08.954412 16319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:34141 every 8 connection(s)
I20260928 10:52:08.956128 16193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:52:08.962914 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 16193
I20260928 10:52:08.963162 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:52:08.963624 6030 external_mini_cluster.cc:1428] Setting key 0af67fd69e9c98f7cb024548fe1b60ea
I20260928 10:52:08.966496 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:08.959034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:57577 (local address 127.5.227.190:34695)
0928 10:52:08.959516 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:08.959521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:08.961040 (+ 1519us) server_negotiation.cc:407] Connection header received
0928 10:52:08.963058 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:08.963065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:08.963145 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:08.963296 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:08.965236 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:08.965242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:08.965246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:08.965263 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:08.965438 (+ 175us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:08.965468 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:08.965590 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:08.966080 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 10:52:08.966163 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":266,"threads_started":1}
I20260928 10:52:08.967505 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:08.959575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:34695 (local address 127.5.227.131:57577)
0928 10:52:08.960863 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:08.960906 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:08.962677 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:08.963599 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:08.963609 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:08.963775 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:08.964850 (+ 1075us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:52:08.964853 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:08.964897 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:52:08.965036 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:08.965808 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:08.965820 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:08.965831 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:52:08.966113 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 10:52:08.966395 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":386,"threads_started":1}
I20260928 10:52:08.968931 16320 heartbeater.cc:347] Connected to a master server at 127.5.227.190:34695
I20260928 10:52:08.969203 16320 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:08.969776 16320 heartbeater.cc:511] Master 127.5.227.190:34695 requested a full tablet report, sending...
I20260928 10:52:08.971210 15872 ts_manager.cc:194] Registered new tserver with Master: 818b0985446840ea91bb73910caf1aaa (127.5.227.131:34141)
I20260928 10:52:08.981057 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:52:08.988234 16322 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:52:08.986316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.5.227.190:34695)
0928 10:52:08.986540 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:08.986576 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:08.987130 (+ 554us) server_negotiation.cc:407] Connection header received
0928 10:52:08.987371 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:08.987375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:08.987427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:08.987537 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:08.987933 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49478: BlockingRecv error: recv got EOF from 127.0.0.1:49478 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 10:52:08.989971 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15931
I20260928 10:52:08.994711 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:08.991219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.5.227.129:45075)
0928 10:52:08.992058 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:08.992067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:08.992102 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:52:08.992171 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:08.992186 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:08.992421 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:08.992570 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:08.993806 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:08.993814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:08.993826 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:08.993842 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:08.993896 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:08.993912 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:08.994071 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:08.994314 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 10:52:08.994407 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":331,"threads_started":1}
I20260928 10:52:08.996076 16053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:09.087658 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15931
I20260928 10:52:09.108513 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 16062
I20260928 10:52:09.113058 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:09.109838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.5.227.130:41299)
0928 10:52:09.110370 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:09.110378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:09.110490 (+ 112us) server_negotiation.cc:407] Connection header received
0928 10:52:09.110566 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:09.110572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:09.110804 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:09.110946 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:09.112240 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:09.112250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:09.112272 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:09.112300 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:09.112353 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:09.112372 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:09.112511 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:09.112679 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 10:52:09.112785 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260928 10:52:09.114898 16184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:09.208884 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 16062
I20260928 10:52:09.227401 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 16193
I20260928 10:52:09.232232 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:09.228657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.5.227.131:34141)
0928 10:52:09.229387 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:09.229394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:09.229426 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:52:09.229487 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:09.229493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:09.229722 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:09.229850 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:09.231080 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:09.231088 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:09.231100 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:52:09.231124 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:09.231175 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:52:09.231191 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:09.231342 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:09.231721 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 10:52:09.231922 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":386,"threads_started":1}
I20260928 10:52:09.233757 16315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:09.330057 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 16193
I20260928 10:52:09.349545 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 15841
I20260928 10:52:09.351281 15902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:09.463856 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 15841
2026-09-28T10:52:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2847 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:52:09 dist-test-slave-8w8g krb5kdc[16339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:52:09 dist-test-slave-8w8g krb5kdc[16339](info): set up 2 sockets
Sep 28 10:52:09 dist-test-slave-8w8g krb5kdc[16339](info): commencing operation
krb5kdc: starting...
W20260928 10:52:10.781921 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:52:10 dist-test-slave-8w8g krb5kdc[16339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:52:11.091372 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:45955
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:45955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:52:11.479596 16352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:11.480005 16352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:11.480248 16352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:11.488426 16352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:52:11.488569 16352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:11.488667 16352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:11.488780 16352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:52:11.488873 16352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:52:11.500135 16352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:45955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:11.501652 16352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:11.503353 16352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:11.519766 16360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:11.519853 16357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:11.519829 16358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:11.520398 16352 server_base.cc:1034] running on GCE node
I20260928 10:52:11.521579 16352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:11.523324 16352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:11.524729 16352 hybrid_clock.cc:648] HybridClock initialized: now 1790592731524547 us; error 62 us; skew 500 ppm
I20260928 10:52:11.525168 16352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:52:11 dist-test-slave-8w8g krb5kdc[16339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592731, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:52:11.528844 16352 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:52:11.530956 16352 webserver.cc:460] Webserver started at http://127.5.227.190:39435/ using document root <none> and password file <none>
I20260928 10:52:11.531968 16352 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:11.532150 16352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:11.532511 16352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:11.536173 16352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "8ecf9193a3a04ff683decae5754baeac"
format_stamp: "Formatted at 2026-09-28 10:52:11 on dist-test-slave-8w8g"
server_key: "8b9523c7a248b9a5dd1cc1c39031859a"
server_key_iv: "eeaf1236831212e7b9d515bbc577e952"
server_key_version: "encryptionkey@0"
I20260928 10:52:11.537144 16352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "8ecf9193a3a04ff683decae5754baeac"
format_stamp: "Formatted at 2026-09-28 10:52:11 on dist-test-slave-8w8g"
server_key: "8b9523c7a248b9a5dd1cc1c39031859a"
server_key_iv: "eeaf1236831212e7b9d515bbc577e952"
server_key_version: "encryptionkey@0"
I20260928 10:52:11.544152 16352 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 10:52:11.548542 16367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:11.550525 16352 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:52:11.550899 16352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "8ecf9193a3a04ff683decae5754baeac"
format_stamp: "Formatted at 2026-09-28 10:52:11 on dist-test-slave-8w8g"
server_key: "8b9523c7a248b9a5dd1cc1c39031859a"
server_key_iv: "eeaf1236831212e7b9d515bbc577e952"
server_key_version: "encryptionkey@0"
I20260928 10:52:11.551247 16352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:11.571931 16352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:11.579173 16352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:11.579525 16352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:11.605443 16352 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:45955
I20260928 10:52:11.605496 16418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:45955 every 8 connection(s)
I20260928 10:52:11.607019 16352 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:52:11.611822 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 16352
I20260928 10:52:11.612058 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:52:11.612715 6030 external_mini_cluster.cc:1428] Setting key a1bf09ed8862938ff736ebe9ba1bafb0
I20260928 10:52:11.612768 16419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:52:11.630851 16419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac: Bootstrap starting.
Sep 28 10:52:11 dist-test-slave-8w8g krb5kdc[16339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:52:11.636581 16419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac: Neither blocks nor log segments found. Creating new log.
I20260928 10:52:11.638254 16419 log.cc:824] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac: Log is configured to *not* fsync() on all Append() calls
I20260928 10:52:11.643429 16419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac: No bootstrap required, opened a new log
I20260928 10:52:11.646437 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:11.615581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.5.227.190:45955)
0928 10:52:11.616527 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:11.616545 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:11.616597 (+ 52us) server_negotiation.cc:407] Connection header received
0928 10:52:11.619028 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:11.619066 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:11.619787 (+ 721us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:11.620345 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:11.622063 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:11.624117 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:11.625905 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:11.627138 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:11.627372 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:11.633813 (+ 6441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:11.633832 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:11.633857 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:11.633925 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:11.639092 (+ 5167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:11.640520 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:11.640531 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:11.640541 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:11.640918 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:11.641497 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:11.641503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:11.641506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:11.642463 (+ 957us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:11.642824 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:11.643855 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0928 10:52:11.644442 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":370,"threads_started":1}
I20260928 10:52:11.654042 16419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf9193a3a04ff683decae5754baeac" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 45955 } }
I20260928 10:52:11.654505 16419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:52:11.654583 16419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecf9193a3a04ff683decae5754baeac, State: Initialized, Role: FOLLOWER
I20260928 10:52:11.655496 16419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf9193a3a04ff683decae5754baeac" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 45955 } }
I20260928 10:52:11.655854 16419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:52:11.656068 16419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:52:11.656304 16419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:52:11.659107 16419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf9193a3a04ff683decae5754baeac" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 45955 } }
I20260928 10:52:11.659711 16419 leader_election.cc:302] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ecf9193a3a04ff683decae5754baeac; no voters:
I20260928 10:52:11.660688 16419 leader_election.cc:288] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:52:11.660821 16424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:52:11.661957 16424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [term 1 LEADER]: Becoming Leader. State: Replica: 8ecf9193a3a04ff683decae5754baeac, State: Running, Role: LEADER
I20260928 10:52:11.662573 16424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf9193a3a04ff683decae5754baeac" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 45955 } }
I20260928 10:52:11.664119 16419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:52:11.666312 16426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ecf9193a3a04ff683decae5754baeac. Latest consensus state: current_term: 1 leader_uuid: "8ecf9193a3a04ff683decae5754baeac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf9193a3a04ff683decae5754baeac" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 45955 } } }
I20260928 10:52:11.666721 16426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [sys.catalog]: This master's current role is: LEADER
I20260928 10:52:11.666597 16425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ecf9193a3a04ff683decae5754baeac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecf9193a3a04ff683decae5754baeac" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 45955 } } }
I20260928 10:52:11.667016 16425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac [sys.catalog]: This master's current role is: LEADER
I20260928 10:52:11.672807 16430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:52:11.679275 16430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:52:11.691932 16430 catalog_manager.cc:1347] Generated new cluster ID: e3ec8a7ba5f8476d88ff038b25394a38
I20260928 10:52:11.692083 16430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:52:11.715107 16430 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:52:11.717419 16430 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:52:11.728643 16430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac: Generated new TSK 0
I20260928 10:52:11.729871 16430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:52:11.805438 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:45955
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:52:12.183990 16445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:12.184336 16445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:12.184501 16445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:12.192351 16445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:12.192490 16445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:12.192687 16445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:52:12.204195 16445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:12.205731 16445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:12.207209 16445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:12.223753 16451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:12.223800 16453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:12.223697 16450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:12.225370 16445 server_base.cc:1034] running on GCE node
I20260928 10:52:12.226136 16445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:12.227278 16445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:12.228525 16445 hybrid_clock.cc:648] HybridClock initialized: now 1790592732228452 us; error 78 us; skew 500 ppm
I20260928 10:52:12.229023 16445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:52:12 dist-test-slave-8w8g krb5kdc[16339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:52:12.232941 16445 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:52:12.234853 16445 webserver.cc:460] Webserver started at http://127.5.227.129:32837/ using document root <none> and password file <none>
I20260928 10:52:12.235637 16445 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:12.235813 16445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:12.236153 16445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:12.239312 16445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "4f6ccbfd9a18406ba3043f576b2ea735"
format_stamp: "Formatted at 2026-09-28 10:52:12 on dist-test-slave-8w8g"
server_key: "d57c8894c8abf1a4916eeb70dfee0ef2"
server_key_iv: "84dedc8037191294db3697c21c4a418c"
server_key_version: "encryptionkey@0"
I20260928 10:52:12.240216 16445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "4f6ccbfd9a18406ba3043f576b2ea735"
format_stamp: "Formatted at 2026-09-28 10:52:12 on dist-test-slave-8w8g"
server_key: "d57c8894c8abf1a4916eeb70dfee0ef2"
server_key_iv: "84dedc8037191294db3697c21c4a418c"
server_key_version: "encryptionkey@0"
I20260928 10:52:12.246518 16445 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260928 10:52:12.251046 16460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:12.253388 16445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:52:12.253784 16445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "4f6ccbfd9a18406ba3043f576b2ea735"
format_stamp: "Formatted at 2026-09-28 10:52:12 on dist-test-slave-8w8g"
server_key: "d57c8894c8abf1a4916eeb70dfee0ef2"
server_key_iv: "84dedc8037191294db3697c21c4a418c"
server_key_version: "encryptionkey@0"
I20260928 10:52:12.254050 16445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:12.271296 16445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:12.277144 16445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:12.277505 16445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:12.278848 16445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:12.280545 16445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:12.280684 16445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:12.280860 16445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:12.280951 16445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:12.320269 16445 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:40423
I20260928 10:52:12.320291 16572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:40423 every 8 connection(s)
I20260928 10:52:12.321794 16445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:52:12.324426 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 16445
I20260928 10:52:12.324648 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:52:12.325716 6030 external_mini_cluster.cc:1428] Setting key ff56a2bee281db8ebb44c15af5c424d8
Sep 28 10:52:12 dist-test-slave-8w8g krb5kdc[16339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:52:12.350445 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:12.325112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:40293 (local address 127.5.227.190:45955)
0928 10:52:12.325725 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:12.325732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:12.326922 (+ 1190us) server_negotiation.cc:407] Connection header received
0928 10:52:12.329697 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:12.329703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:12.329790 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:12.329894 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:12.332607 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.333408 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.336260 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.336735 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.336866 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:12.341331 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:12.341341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:12.341348 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:12.341411 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:12.346156 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:12.347219 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:12.347229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:12.347236 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:12.347421 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:12.348036 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:12.348044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:12.348050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:12.348484 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:12.348654 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:12.349675 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0928 10:52:12.349961 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":342,"threads_started":1}
I20260928 10:52:12.351291 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:12.325796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:45955 (local address 127.5.227.129:40293)
0928 10:52:12.326730 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:12.326783 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:12.329252 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:12.330241 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:12.330256 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:12.331268 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:12.332359 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:12.332389 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.333598 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.333606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:12.336012 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:12.336036 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.337034 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.337041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:12.337269 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:12.338256 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:12.338282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:12.340983 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:12.346405 (+ 5422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:52:12.346415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:52:12.346431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:52:12.347030 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:52:12.347550 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:52:12.347553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:52:12.347556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:52:12.347870 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:52:12.348665 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:12.348674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:12.349214 (+ 540us) client_negotiation.cc:769] Sending connection context
0928 10:52:12.349496 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 10:52:12.349978 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":381,"threads_started":1}
I20260928 10:52:12.353348 16573 heartbeater.cc:347] Connected to a master server at 127.5.227.190:45955
I20260928 10:52:12.353777 16573 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:12.354674 16573 heartbeater.cc:511] Master 127.5.227.190:45955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:52:12.357540 16384 ts_manager.cc:194] Registered new tserver with Master: 4f6ccbfd9a18406ba3043f576b2ea735 (127.5.227.129:40423)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:52:12.359901 16384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:40293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:52:12.400997 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:45955
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:52:12.769034 16579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:12.769356 16579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:12.769451 16579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:12.778172 16579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:12.778322 16579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:12.778440 16579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:52:12.789220 16579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:12.790594 16579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:12.792410 16579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:12.808719 16584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:12.808799 16587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:12.808792 16585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:12.809870 16579 server_base.cc:1034] running on GCE node
I20260928 10:52:12.810745 16579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:12.811987 16579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:12.813323 16579 hybrid_clock.cc:648] HybridClock initialized: now 1790592732813321 us; error 150 us; skew 500 ppm
I20260928 10:52:12.813716 16579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:52:12 dist-test-slave-8w8g krb5kdc[16339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:52:12.817443 16579 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:52:12.819600 16579 webserver.cc:460] Webserver started at http://127.5.227.130:38125/ using document root <none> and password file <none>
I20260928 10:52:12.820418 16579 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:12.820534 16579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:12.820953 16579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:12.824216 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "5fc3b376fd4c40a2891945c226ff2beb"
format_stamp: "Formatted at 2026-09-28 10:52:12 on dist-test-slave-8w8g"
server_key: "a64a0df3b04592286e163bcfe6f906ee"
server_key_iv: "64969a3c521da50c9c1205bdcac2c700"
server_key_version: "encryptionkey@0"
I20260928 10:52:12.825031 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "5fc3b376fd4c40a2891945c226ff2beb"
format_stamp: "Formatted at 2026-09-28 10:52:12 on dist-test-slave-8w8g"
server_key: "a64a0df3b04592286e163bcfe6f906ee"
server_key_iv: "64969a3c521da50c9c1205bdcac2c700"
server_key_version: "encryptionkey@0"
I20260928 10:52:12.830978 16579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:52:12.835299 16594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:12.837700 16579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:52:12.837920 16579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "5fc3b376fd4c40a2891945c226ff2beb"
format_stamp: "Formatted at 2026-09-28 10:52:12 on dist-test-slave-8w8g"
server_key: "a64a0df3b04592286e163bcfe6f906ee"
server_key_iv: "64969a3c521da50c9c1205bdcac2c700"
server_key_version: "encryptionkey@0"
I20260928 10:52:12.838169 16579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:12.878554 16579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:12.884354 16579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:12.884622 16579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:12.885679 16579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:12.887274 16579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:12.887356 16579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:12.887452 16579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:12.887497 16579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:12.930572 16579 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:45263
I20260928 10:52:12.930629 16706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:45263 every 8 connection(s)
I20260928 10:52:12.932024 16579 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:52:12.939853 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 16579
I20260928 10:52:12.940055 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:52:12.940631 6030 external_mini_cluster.cc:1428] Setting key 8c6027d99a6fb802443c11e5ccd32cc4
Sep 28 10:52:12 dist-test-slave-8w8g krb5kdc[16339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592732, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:52:12.958316 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:12.935068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:59157 (local address 127.5.227.190:45955)
0928 10:52:12.935854 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:12.935863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:12.936461 (+ 598us) server_negotiation.cc:407] Connection header received
0928 10:52:12.938393 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:12.938398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:12.938520 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:12.938659 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:12.940877 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.942012 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.944774 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.945268 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.945383 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:12.950321 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:12.950328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:12.950331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:12.950369 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:12.953455 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:12.954910 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:12.954916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:12.954918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:12.955084 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:12.955883 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:12.955891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:12.955895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:12.956256 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:12.956384 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:12.957720 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0928 10:52:12.957927 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":324,"threads_started":1}
I20260928 10:52:12.959843 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:12.935476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:45955 (local address 127.5.227.130:59157)
0928 10:52:12.936288 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:12.936332 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:12.938071 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:12.938946 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:12.938955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:12.939670 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:12.940592 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:12.940627 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.942327 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.942333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:12.944563 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:12.944587 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:12.945542 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:12.945553 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:12.945918 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:12.946792 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:12.946827 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:12.949982 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:12.953861 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:52:12.953876 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:52:12.953900 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:52:12.954674 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:52:12.955223 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:52:12.955231 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:52:12.955237 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:52:12.955514 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:52:12.956403 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:12.956415 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:12.957190 (+ 775us) client_negotiation.cc:769] Sending connection context
0928 10:52:12.957532 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 10:52:12.958214 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":377,"threads_started":1}
I20260928 10:52:12.961709 16707 heartbeater.cc:347] Connected to a master server at 127.5.227.190:45955
I20260928 10:52:12.962083 16707 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:12.962940 16707 heartbeater.cc:511] Master 127.5.227.190:45955 requested a full tablet report, sending...
I20260928 10:52:12.964795 16384 ts_manager.cc:194] Registered new tserver with Master: 5fc3b376fd4c40a2891945c226ff2beb (127.5.227.130:45263)
I20260928 10:52:12.966189 16384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:59157
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:52:13.014874 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:45955
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:52:13.363806 16573 heartbeater.cc:503] Master 127.5.227.190:45955 was elected leader, sending a full tablet report...
W20260928 10:52:13.388320 16713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:52:13.388684 16713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:52:13.388855 16713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:52:13.397555 16713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:52:13.397759 16713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:52:13.397955 16713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:52:13.408754 16713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:52:13.410174 16713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:52:13.411602 16713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:52:13.427563 16718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:13.427598 16719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:52:13.427627 16721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:52:13.428313 16713 server_base.cc:1034] running on GCE node
I20260928 10:52:13.429127 16713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:52:13.430132 16713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:52:13.431384 16713 hybrid_clock.cc:648] HybridClock initialized: now 1790592733431337 us; error 29 us; skew 500 ppm
I20260928 10:52:13.431947 16713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:52:13 dist-test-slave-8w8g krb5kdc[16339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592733, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:52:13.436040 16713 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:52:13.438385 16713 webserver.cc:460] Webserver started at http://127.5.227.131:42931/ using document root <none> and password file <none>
I20260928 10:52:13.439775 16713 fs_manager.cc:359] Metadata directory not provided
I20260928 10:52:13.439941 16713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:52:13.440310 16713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:52:13.444783 16713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "d712d0f4088d479cb8ffe6c748ccd534"
format_stamp: "Formatted at 2026-09-28 10:52:13 on dist-test-slave-8w8g"
server_key: "e481cc42cb6b963837731106396ed70c"
server_key_iv: "ff72224c37e1189294194a5b246de84d"
server_key_version: "encryptionkey@0"
I20260928 10:52:13.446007 16713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "d712d0f4088d479cb8ffe6c748ccd534"
format_stamp: "Formatted at 2026-09-28 10:52:13 on dist-test-slave-8w8g"
server_key: "e481cc42cb6b963837731106396ed70c"
server_key_iv: "ff72224c37e1189294194a5b246de84d"
server_key_version: "encryptionkey@0"
I20260928 10:52:13.453913 16713 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260928 10:52:13.458998 16728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:13.461010 16713 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:52:13.461338 16713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "d712d0f4088d479cb8ffe6c748ccd534"
format_stamp: "Formatted at 2026-09-28 10:52:13 on dist-test-slave-8w8g"
server_key: "e481cc42cb6b963837731106396ed70c"
server_key_iv: "ff72224c37e1189294194a5b246de84d"
server_key_version: "encryptionkey@0"
I20260928 10:52:13.461782 16713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:52:13.504631 16713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:52:13.510341 16713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:52:13.510681 16713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:52:13.511859 16713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:52:13.513330 16713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:52:13.513418 16713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:13.513551 16713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:52:13.513612 16713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:52:13.556145 16713 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:44755
I20260928 10:52:13.556188 16840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:44755 every 8 connection(s)
I20260928 10:52:13.557982 16713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:52:13.563795 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 16713
I20260928 10:52:13.563992 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:52:13.564491 6030 external_mini_cluster.cc:1428] Setting key ceabe668e141bc121d593b2c1344fd26
Sep 28 10:52:13 dist-test-slave-8w8g krb5kdc[16339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592733, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:52:13.588830 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.561433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:36237 (local address 127.5.227.190:45955)
0928 10:52:13.561990 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.561995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.563192 (+ 1197us) server_negotiation.cc:407] Connection header received
0928 10:52:13.565635 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.565641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.565743 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.565843 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:13.568300 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.569481 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.573216 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.574362 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.574560 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.580438 (+ 5878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:13.580447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:13.580451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:13.580514 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:13.584730 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:13.585653 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:13.585661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:13.585668 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:13.585797 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:13.586280 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:13.586287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:13.586295 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:13.586594 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:13.587102 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.588030 (+ 928us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.588299 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
I20260928 10:52:13.589887 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.562124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:45955 (local address 127.5.227.131:36237)
0928 10:52:13.562984 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:13.563030 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:13.565189 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:13.566363 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:13.566376 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:13.567050 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:52:13.568040 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.568068 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.569797 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.569806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.572758 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.572787 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.574725 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.574736 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.575042 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.576022 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:52:13.576049 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:52:13.579986 (+ 3937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:52:13.584933 (+ 4947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:52:13.584945 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:52:13.584972 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:52:13.585485 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:52:13.585923 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:52:13.585929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:52:13.585933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:52:13.586139 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:52:13.587018 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:52:13.587029 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:52:13.587476 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 10:52:13.587892 (+ 416us) client_negotiation.cc:240] Negotiation successful
0928 10:52:13.588514 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":435,"threads_started":1}
I20260928 10:52:13.592547 16841 heartbeater.cc:347] Connected to a master server at 127.5.227.190:45955
I20260928 10:52:13.593093 16841 heartbeater.cc:464] Registering TS with master...
I20260928 10:52:13.594846 16841 heartbeater.cc:511] Master 127.5.227.190:45955 requested a full tablet report, sending...
I20260928 10:52:13.597173 16384 ts_manager.cc:194] Registered new tserver with Master: d712d0f4088d479cb8ffe6c748ccd534 (127.5.227.131:44755)
I20260928 10:52:13.598039 16384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:36237
I20260928 10:52:13.610033 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:52:13.636044 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.619361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.5.227.190:45955)
0928 10:52:13.619549 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.619554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.620930 (+ 1376us) server_negotiation.cc:407] Connection header received
0928 10:52:13.621610 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.621620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.621708 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.621826 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:13.624052 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.624841 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.626210 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.626626 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.626778 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.629364 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:13.629370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:13.629373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:13.629414 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:13.632737 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:13.633681 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:13.633687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:13.633690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:13.633834 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:13.634302 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:13.634308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:13.634310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:13.634707 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:13.634979 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.635488 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.635788 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:52:13.641523 16384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:52:13.649242 16384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:52:13.686041 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.666064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.5.227.129:40423)
0928 10:52:13.676598 (+ 10534us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.676607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.676636 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:52:13.676813 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.676820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.677024 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.677234 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:52:13.678574 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.679587 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.682734 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.684375 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.684518 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.684759 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.685523 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.685709 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10375,"thread_start_us":10105,"threads_started":1}
I20260928 10:52:13.686156 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.665710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48708)
0928 10:52:13.674726 (+ 9016us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:13.674754 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:13.674895 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:13.677486 (+ 2591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:13.677491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:13.677519 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:52:13.678337 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.678349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.679763 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.679771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.682401 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.682419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.684907 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.684914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.685161 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.685297 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 10:52:13.685460 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 10:52:13.685667 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8826,"mutex_wait_us":780,"thread_start_us":7984,"threads_started":1}
I20260928 10:52:13.686277 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.665362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:45263 (local address 127.0.0.1:47860)
0928 10:52:13.666096 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:13.666191 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:13.667284 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:13.677023 (+ 9739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:13.677031 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:13.677095 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:52:13.678347 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.678356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.680215 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.680220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.682406 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.682419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.684459 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.684466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.685218 (+ 752us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.685299 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 10:52:13.685445 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 10:52:13.685797 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":801,"spinlock_wait_cycles":1920,"thread_start_us":899,"threads_started":1}
I20260928 10:52:13.686580 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.664845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.5.227.131:44755)
0928 10:52:13.676324 (+ 11479us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.676333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.676369 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:52:13.676464 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.676471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.676792 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.678032 (+ 1240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:52:13.678256 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.679916 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.682803 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.684375 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.684813 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.685209 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.685763 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.685971 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11246,"thread_start_us":10948,"threads_started":1}
I20260928 10:52:13.686952 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.664459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:44755 (local address 127.0.0.1:36630)
0928 10:52:13.666096 (+ 1637us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:13.666273 (+ 177us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:13.673908 (+ 7635us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x61200002c628. stack: 00007fefacb0166e 00007fefacb01430 00007fefabb885ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fefacaf70fd 00007fefacaf6d1e 00007fefbfa003cd 00007fefbfa0036c 00007fefbf9ccb3b 00007fefb3506536 00007fefb3504d9e 00007fefb358cdca 00007fefb358af68 00007fefb35be82f
0928 10:52:13.673976 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:13.677019 (+ 3043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:13.677033 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:13.677077 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:52:13.677799 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.677819 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.680142 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.680148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.682492 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.682506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.684553 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.684558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.685108 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.685299 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 10:52:13.685442 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 10:52:13.686585 (+ 1143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"mutex_wait_us":1284,"spinlock_wait_cycles":2864768,"thread_start_us":2797,"threads_started":1}
I20260928 10:52:13.687718 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.666056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.5.227.130:45263)
0928 10:52:13.675907 (+ 9851us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.675919 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.675954 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:52:13.676075 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.676083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.676727 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.677238 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:52:13.678532 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.680051 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.682616 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.684208 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.684405 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.684879 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.687202 (+ 2323us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.687411 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9341,"thread_start_us":9194,"threads_started":1}
I20260928 10:52:13.690827 16508 tablet_service.cc:1467] Processing CreateTablet for tablet cb336a1180434fc4bbe873ca2e18fba9 (DEFAULT_TABLE table=test-table [id=7ee13dfe456e4066a184760592500c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:52:13.691294 16776 tablet_service.cc:1467] Processing CreateTablet for tablet cb336a1180434fc4bbe873ca2e18fba9 (DEFAULT_TABLE table=test-table [id=7ee13dfe456e4066a184760592500c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:52:13.691447 16642 tablet_service.cc:1467] Processing CreateTablet for tablet cb336a1180434fc4bbe873ca2e18fba9 (DEFAULT_TABLE table=test-table [id=7ee13dfe456e4066a184760592500c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:52:13.692462 16508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb336a1180434fc4bbe873ca2e18fba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:52:13.693113 16776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb336a1180434fc4bbe873ca2e18fba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:52:13.693154 16642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb336a1180434fc4bbe873ca2e18fba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:52:13.709623 16860 tablet_bootstrap.cc:492] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Bootstrap starting.
I20260928 10:52:13.713503 16860 tablet_bootstrap.cc:654] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Neither blocks nor log segments found. Creating new log.
I20260928 10:52:13.714900 16860 log.cc:824] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Log is configured to *not* fsync() on all Append() calls
I20260928 10:52:13.716943 16862 tablet_bootstrap.cc:492] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: Bootstrap starting.
I20260928 10:52:13.718019 16861 tablet_bootstrap.cc:492] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Bootstrap starting.
I20260928 10:52:13.721505 16860 tablet_bootstrap.cc:492] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: No bootstrap required, opened a new log
I20260928 10:52:13.721894 16860 ts_tablet_manager.cc:1397] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260928 10:52:13.722141 16861 tablet_bootstrap.cc:654] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Neither blocks nor log segments found. Creating new log.
I20260928 10:52:13.722712 16862 tablet_bootstrap.cc:654] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: Neither blocks nor log segments found. Creating new log.
I20260928 10:52:13.723706 16861 log.cc:824] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Log is configured to *not* fsync() on all Append() calls
I20260928 10:52:13.724285 16862 log.cc:824] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: Log is configured to *not* fsync() on all Append() calls
I20260928 10:52:13.727190 16861 tablet_bootstrap.cc:492] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: No bootstrap required, opened a new log
I20260928 10:52:13.727502 16862 tablet_bootstrap.cc:492] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: No bootstrap required, opened a new log
I20260928 10:52:13.727531 16861 ts_tablet_manager.cc:1397] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260928 10:52:13.727847 16862 ts_tablet_manager.cc:1397] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 10:52:13.733129 16860 raft_consensus.cc:348] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.734046 16860 raft_consensus.cc:374] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:52:13.734207 16860 raft_consensus.cc:729] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d712d0f4088d479cb8ffe6c748ccd534, State: Initialized, Role: FOLLOWER
I20260928 10:52:13.735069 16860 consensus_queue.cc:260] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.736832 16841 heartbeater.cc:503] Master 127.5.227.190:45955 was elected leader, sending a full tablet report...
I20260928 10:52:13.737823 16860 ts_tablet_manager.cc:1428] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260928 10:52:13.737720 16861 raft_consensus.cc:348] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.738211 16861 raft_consensus.cc:374] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:52:13.738314 16861 raft_consensus.cc:729] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc3b376fd4c40a2891945c226ff2beb, State: Initialized, Role: FOLLOWER
I20260928 10:52:13.739151 16861 consensus_queue.cc:260] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.740553 16707 heartbeater.cc:503] Master 127.5.227.190:45955 was elected leader, sending a full tablet report...
I20260928 10:52:13.740410 16862 raft_consensus.cc:348] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.741129 16862 raft_consensus.cc:374] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:52:13.741261 16862 raft_consensus.cc:729] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6ccbfd9a18406ba3043f576b2ea735, State: Initialized, Role: FOLLOWER
I20260928 10:52:13.742275 16862 consensus_queue.cc:260] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.743017 16861 ts_tablet_manager.cc:1428] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260928 10:52:13.745327 16862 ts_tablet_manager.cc:1428] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260928 10:52:13.810230 16842 tablet.cc:2369] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:52:13.824497 16574 tablet.cc:2369] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:52:13.844069 16868 raft_consensus.cc:482] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:52:13.844472 16868 raft_consensus.cc:504] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.846300 16868 leader_election.cc:288] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d712d0f4088d479cb8ffe6c748ccd534 (127.5.227.131:44755), 5fc3b376fd4c40a2891945c226ff2beb (127.5.227.130:45263)
I20260928 10:52:13.858176 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.847714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:50073 (local address 127.5.227.130:45263)
0928 10:52:13.847959 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.847967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.848236 (+ 269us) server_negotiation.cc:407] Connection header received
0928 10:52:13.848328 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.848332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.848396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.848500 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:52:13.849331 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.850547 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.854940 (+ 4393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.856471 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.856632 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.856739 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.857716 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.857878 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:52:13.858186 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.847369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:41963 (local address 127.5.227.131:44755)
0928 10:52:13.847593 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.847601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.847689 (+ 88us) server_negotiation.cc:407] Connection header received
0928 10:52:13.848058 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.848066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.848163 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.848235 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:52:13.849140 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.850557 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.854929 (+ 4372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.856413 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.856644 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.856739 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.857716 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.857878 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:52:13.858800 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.846804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:44755 (local address 127.5.227.129:41963)
0928 10:52:13.847538 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:13.847560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:13.847853 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:13.848430 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:13.848433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:13.848452 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:52:13.848934 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.848953 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.850778 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.850784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.854513 (+ 3729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.854538 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.856986 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.856999 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.857225 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.857318 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 10:52:13.857586 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 10:52:13.857836 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":332,"thread_start_us":397,"threads_started":1}
I20260928 10:52:13.858834 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.846910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:45263 (local address 127.5.227.129:50073)
0928 10:52:13.848058 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:13.848078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:52:13.848203 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:52:13.848539 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:52:13.848543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:52:13.848583 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:52:13.849141 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.849158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.850779 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.850784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.854513 (+ 3729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:52:13.854538 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.857010 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.857015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:52:13.857219 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.857319 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 10:52:13.857583 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 10:52:13.857810 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":568,"spinlock_wait_cycles":28416,"thread_start_us":859,"threads_started":1}
I20260928 10:52:13.860284 16796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb336a1180434fc4bbe873ca2e18fba9" candidate_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d712d0f4088d479cb8ffe6c748ccd534" is_pre_election: true
I20260928 10:52:13.860373 16662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb336a1180434fc4bbe873ca2e18fba9" candidate_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc3b376fd4c40a2891945c226ff2beb" is_pre_election: true
I20260928 10:52:13.861016 16796 raft_consensus.cc:2454] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f6ccbfd9a18406ba3043f576b2ea735 in term 0.
I20260928 10:52:13.861012 16662 raft_consensus.cc:2454] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f6ccbfd9a18406ba3043f576b2ea735 in term 0.
I20260928 10:52:13.861930 16464 leader_election.cc:302] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f6ccbfd9a18406ba3043f576b2ea735, 5fc3b376fd4c40a2891945c226ff2beb, d712d0f4088d479cb8ffe6c748ccd534; no voters:
I20260928 10:52:13.862638 16868 raft_consensus.cc:2781] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:52:13.862802 16868 raft_consensus.cc:482] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:52:13.863143 16868 raft_consensus.cc:3037] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:52:13.866039 16868 raft_consensus.cc:504] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.866755 16868 leader_election.cc:288] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [CANDIDATE]: Term 1 election: Requested vote from peers d712d0f4088d479cb8ffe6c748ccd534 (127.5.227.131:44755), 5fc3b376fd4c40a2891945c226ff2beb (127.5.227.130:45263)
I20260928 10:52:13.867375 16796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb336a1180434fc4bbe873ca2e18fba9" candidate_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d712d0f4088d479cb8ffe6c748ccd534"
I20260928 10:52:13.867408 16662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb336a1180434fc4bbe873ca2e18fba9" candidate_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc3b376fd4c40a2891945c226ff2beb"
I20260928 10:52:13.867627 16662 raft_consensus.cc:3037] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:52:13.867787 16796 raft_consensus.cc:3037] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:52:13.869997 16662 raft_consensus.cc:2454] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6ccbfd9a18406ba3043f576b2ea735 in term 1.
I20260928 10:52:13.870487 16796 raft_consensus.cc:2454] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6ccbfd9a18406ba3043f576b2ea735 in term 1.
I20260928 10:52:13.870518 16463 leader_election.cc:302] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f6ccbfd9a18406ba3043f576b2ea735, 5fc3b376fd4c40a2891945c226ff2beb; no voters:
I20260928 10:52:13.870877 16868 raft_consensus.cc:2781] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:52:13.871380 16868 raft_consensus.cc:686] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 1 LEADER]: Becoming Leader. State: Replica: 4f6ccbfd9a18406ba3043f576b2ea735, State: Running, Role: LEADER
I20260928 10:52:13.871974 16868 consensus_queue.cc:237] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } }
I20260928 10:52:13.878898 16384 catalog_manager.cc:5495] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f6ccbfd9a18406ba3043f576b2ea735 (127.5.227.129). New cstate: current_term: 1 leader_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6ccbfd9a18406ba3043f576b2ea735" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40423 } health_report { overall_health: HEALTHY } } }
I20260928 10:52:13.922834 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.916106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.5.227.129:40423)
0928 10:52:13.916324 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.916329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.916349 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:52:13.916551 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.916563 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.916741 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.916920 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:52:13.917463 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.918365 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.920235 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.920657 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.920775 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:13.921399 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:52:13.921984 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:52:13.922111 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:13.922268 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 10:52:13.922347 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:52:13.932358 16662 raft_consensus.cc:1261] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 1 FOLLOWER]: Refusing update from remote peer 4f6ccbfd9a18406ba3043f576b2ea735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:52:13.932407 16796 raft_consensus.cc:1261] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 1 FOLLOWER]: Refusing update from remote peer 4f6ccbfd9a18406ba3043f576b2ea735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:52:13.933553 16871 consensus_queue.cc:1035] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d712d0f4088d479cb8ffe6c748ccd534" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 44755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:52:13.934219 16868 consensus_queue.cc:1035] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc3b376fd4c40a2891945c226ff2beb" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:52:13.934865 16708 tablet.cc:2369] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:52:13.957243 16875 mvcc.cc:205] Tried to move back new op lower bound from 7334267838175367168 to 7334267837945581568. Current Snapshot: MvccSnapshot[applied={T|T < 7334267838175367168}]
I20260928 10:52:13.969318 16383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:52:13.969729 16383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:52:13.975977 16383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ecf9193a3a04ff683decae5754baeac: Sending DeleteTablet for 3 replicas of tablet cb336a1180434fc4bbe873ca2e18fba9
I20260928 10:52:13.977432 16776 tablet_service.cc:1514] Processing DeleteTablet for tablet cb336a1180434fc4bbe873ca2e18fba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:52:13 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:36630
I20260928 10:52:13.977785 16642 tablet_service.cc:1514] Processing DeleteTablet for tablet cb336a1180434fc4bbe873ca2e18fba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:52:13 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:47860
I20260928 10:52:13.978040 16508 tablet_service.cc:1514] Processing DeleteTablet for tablet cb336a1180434fc4bbe873ca2e18fba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:52:13 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:48708
I20260928 10:52:13.980049 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 16445
I20260928 10:52:13.998375 16884 tablet_replica.cc:330] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: stopping tablet replica
I20260928 10:52:13.999435 16885 tablet_replica.cc:330] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: stopping tablet replica
I20260928 10:52:13.999890 16885 raft_consensus.cc:2229] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:52:13.999924 16884 raft_consensus.cc:2229] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:52:14.000422 16885 raft_consensus.cc:2258] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:52:14.000442 16884 raft_consensus.cc:2258] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:52:14.002414 16886 tablet_replica.cc:330] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: stopping tablet replica
I20260928 10:52:14.003077 16884 ts_tablet_manager.cc:1905] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:52:14.003400 16886 raft_consensus.cc:2229] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:52:14.004233 16886 raft_consensus.cc:2258] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:52:14.006238 16885 ts_tablet_manager.cc:1905] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 10:52:14 dist-test-slave-8w8g krb5kdc[16339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:52:14.008715 16886 ts_tablet_manager.cc:1905] T cb336a1180434fc4bbe873ca2e18fba9 P 4f6ccbfd9a18406ba3043f576b2ea735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:52:14.011401 16884 ts_tablet_manager.cc:1918] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:52:14.011636 16884 log.cc:1196] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/cb336a1180434fc4bbe873ca2e18fba9
I20260928 10:52:14.012262 16884 ts_tablet_manager.cc:1939] T cb336a1180434fc4bbe873ca2e18fba9 P d712d0f4088d479cb8ffe6c748ccd534: Deleting consensus metadata
I20260928 10:52:14.014581 16371 catalog_manager.cc:4841] TS d712d0f4088d479cb8ffe6c748ccd534 (127.5.227.131:44755): tablet cb336a1180434fc4bbe873ca2e18fba9 (table test-table [id=7ee13dfe456e4066a184760592500c10]) successfully deleted
I20260928 10:52:14.016660 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:13.982555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.5.227.129:40423)
0928 10:52:13.982817 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:13.982827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:13.986266 (+ 3439us) server_negotiation.cc:407] Connection header received
0928 10:52:13.987111 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:13.987122 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:13.987278 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:13.987386 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:13.992224 (+ 4838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:13.993935 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:13.999403 (+ 5468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:14.001524 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:14.001897 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:14.008263 (+ 6366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:14.008280 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:14.008300 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:14.008359 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:14.012799 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:14.013984 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:14.014001 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:14.014004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:14.014258 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:14.014774 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:14.014779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:14.014783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:14.015237 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:14.015598 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:14.015927 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 10:52:14.016283 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:52:14.017527 16568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:14.021224 16885 ts_tablet_manager.cc:1918] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:52:14.021446 16885 log.cc:1196] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/cb336a1180434fc4bbe873ca2e18fba9
I20260928 10:52:14.022081 16885 ts_tablet_manager.cc:1939] T cb336a1180434fc4bbe873ca2e18fba9 P 5fc3b376fd4c40a2891945c226ff2beb: Deleting consensus metadata
I20260928 10:52:14.025170 16370 catalog_manager.cc:4841] TS 5fc3b376fd4c40a2891945c226ff2beb (127.5.227.130:45263): tablet cb336a1180434fc4bbe873ca2e18fba9 (table test-table [id=7ee13dfe456e4066a184760592500c10]) successfully deleted
I20260928 10:52:14.126652 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 16445
W20260928 10:52:14.145864 16368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.227.129:40423 (error 108)
I20260928 10:52:14.146732 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 16579
I20260928 10:52:14.147822 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.147284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48734)
0928 10:52:14.147479 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:14.147607 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 10:52:14.148077 16368 catalog_manager.cc:4584] TS 4f6ccbfd9a18406ba3043f576b2ea735 (127.5.227.129:40423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb336a1180434fc4bbe873ca2e18fba9: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Sep 28 10:52:14 dist-test-slave-8w8g krb5kdc[16339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:52:14.165283 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.147888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.5.227.130:45263)
0928 10:52:14.148086 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:14.148091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:14.148106 (+ 15us) server_negotiation.cc:407] Connection header received
0928 10:52:14.148157 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:14.148161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:14.148233 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:14.148362 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:14.149671 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:14.150419 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:14.152273 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:14.152891 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:14.153018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:14.158161 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:14.158178 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:14.158195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:14.158246 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:14.161682 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:14.162740 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:14.162749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:14.162751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:14.162892 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:14.163365 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:14.163369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:14.163371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:14.163923 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:14.164240 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:14.164664 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 10:52:14.164848 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:52:14.166199 16702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:14.202271 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.201605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48740)
0928 10:52:14.201932 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:14.202066 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 10:52:14.203843 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.203265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48750)
0928 10:52:14.203486 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:14.203589 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 10:52:14.261791 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 16579
I20260928 10:52:14.264439 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.263719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48762)
0928 10:52:14.264051 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:14.264211 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 10:52:14.266306 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.265859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48774)
0928 10:52:14.266051 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:14.266167 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 10:52:14.280840 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 16713
Sep 28 10:52:14 dist-test-slave-8w8g krb5kdc[16339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:52:14.298209 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.281898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.5.227.131:44755)
0928 10:52:14.282121 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:52:14.282126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:52:14.282242 (+ 116us) server_negotiation.cc:407] Connection header received
0928 10:52:14.282423 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:52:14.282427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:52:14.282493 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:52:14.282604 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:52:14.284150 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:14.284889 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:14.286226 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:52:14.286647 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:52:14.286753 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:52:14.291941 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:52:14.291957 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:52:14.291974 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:52:14.292064 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:52:14.295208 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:14.296083 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:14.296096 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:14.296099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:14.296251 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:52:14.296659 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:52:14.296662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:52:14.296664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:52:14.297049 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:52:14.297333 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 10:52:14.297709 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 10:52:14.297878 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:52:14.299005 16836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:14.347229 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.346584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48778)
0928 10:52:14.346918 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:14.347047 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 10:52:14.348595 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:52:14.348127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40423 (local address 127.0.0.1:48790)
0928 10:52:14.348340 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 10:52:14.348426 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 10:52:14.396087 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 16713
I20260928 10:52:14.414017 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 16352
I20260928 10:52:14.415354 16414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:52:14.533645 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 16352
2026-09-28T10:52:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5070 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:52:14 dist-test-slave-8w8g krb5kdc[16895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:52:14 dist-test-slave-8w8g krb5kdc[16895](info): set up 2 sockets
Sep 28 10:52:14 dist-test-slave-8w8g krb5kdc[16895](info): commencing operation
krb5kdc: starting...
W20260928 10:52:15.821473 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:52:16 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:52:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/rangeradmin_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/rangeradmin_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/rangerlookup_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/rangerlookup_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
I20260928 10:52:16.237124 6030 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko9h9KU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/postgres -l logfile start
2026-09-28 10:52:20.406 UTC [16925] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 10:52:20.407 UTC [16925] LOG: listening on IPv4 address "127.5.227.148", port 40777
2026-09-28 10:52:20.415 UTC [16925] LOG: listening on Unix socket "/tmp/.s.PGSQL.40777"
2026-09-28 10:52:20.442 UTC [16928] LOG: database system was shut down at 2026-09-28 10:52:17 UTC
2026-09-28 10:52:20.448 UTC [16925] LOG: database system is ready to accept connections
I20260928 10:52:21.579166 6030 mini_postgres.cc:96] Postgres bound to 40777
2026-09-28 10:52:21.599 UTC [16936] FATAL: database "slave" does not exist
127.5.227.148:40777 - accepting connections
I20260928 10:52:21.600147 6030 mini_ranger.cc:161] Starting Ranger...
I20260928 10:52:21.633011 6030 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 10:52:22.510721 6030 mini_ranger.cc:87] Created ranger Postgres database
I20260928 10:52:22.510946 6030 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-admin
2026-09-28 10:52:22,857 [I] DB FLAVOR :POSTGRES
2026-09-28 10:52:22,857 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:52:22,857 [I] Checking connection..
2026-09-28 10:52:22,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 10:52:23,158 [I] Checking connection passed.
2026-09-28 10:52:23,158 [I] --------- Verifying version history table ---------
2026-09-28 10:52:23,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:52:23,458 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:52:23,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:52:23,775 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:52:23,776 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 10:52:23,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 10:52:24.073 UTC [17000] WARNING: there is no transaction in progress
2026-09-28 10:52:24,080 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 10:52:24,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:52:24,382 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 10:52:24,383 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:52:24,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 10:52:24,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 10:52:24,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8w8g.c.gcp-upstream.internal','N') ;"
2026-09-28 10:52:25,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:52:25,474 [I] Table x_portal_user does not exist in database ranger
2026-09-28 10:52:25,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:52:25,751 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 10:52:25,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:52:26,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:52:26,283 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 10:52:26,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 10:52:27.639 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.662 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.685 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.708 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.730 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.753 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.827 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.838 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.859 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.874 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.890 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.910 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.928 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.940 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.953 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:27.965 UTC [17108] WARNING: there is no transaction in progress
2026-09-28 10:52:29,167 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 10:52:29,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 10:52:29,480 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 10:52:29,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:52:29,798 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 10:52:29,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:52:30,123 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 10:52:30,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:52:30,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:52:30,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8w8g.c.gcp-upstream.internal';"
2026-09-28 10:52:31,052 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 10:52:31,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 10:52:31,329 [I] DB_PATCHES have already been applied
I20260928 10:52:31.336310 6030 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko9h9KU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 10:52:31.336475 6030 mini_ranger.cc:193] Using host: 127.5.227.148
I20260928 10:52:31.354266 6030 mini_ranger.cc:227] Ranger admin URL: http://127.5.227.148:49939
Sep 28, 2026 10:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:52:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:52:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 10:52:32.703712 6030 mini_ranger.cc:160] Time spent starting Ranger: real 11.104s user 0.000s sys 0.048s
Sep 28, 2026 10:52:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:52:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 10:52:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:52:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:52:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 10:53:00.209321 6030 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/rangerkms_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/rangerkms_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.227.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.227.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 10:53:00 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592780, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 10:53:00.439960 6030 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 10:53:00.467770 6030 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 10:53:01.912456 6030 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 10:53:01.912638 6030 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms
I20260928 10:53:01.912812 6030 mini_ranger_kms.cc:227] Using postgres at 127.5.227.148:40777
2026-09-28 10:53:02,442 [I] DB FLAVOR :POSTGRES
2026-09-28 10:53:02,442 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:53:02,443 [I] Checking connection
2026-09-28 10:53:02,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 10:53:02,755 [I] connection success
2026-09-28 10:53:02,756 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 10:53:02,756 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 10:53:02,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 10:53:03,071 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 10:53:03,071 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:53:03,071 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 10:53:03,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:40777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 10:53:03,438 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 10:53:04.103540 6030 mini_ranger_kms.cc:310] Created kms service
I20260928 10:53:04.375537 6030 mini_ranger_kms.cc:326] Created kudu user
I20260928 10:53:04.431785 6030 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 10:53:04 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592780, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:53:05.009670 6030 mini_ranger_kms.cc:384] Added ranger policy
I20260928 10:53:05.010039 6030 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 10:53:05.010113 6030 mini_ranger_kms.cc:240] Using host: 127.5.227.148
I20260928 10:53:05.036480 6030 mini_ranger_kms.cc:276] Ranger KMS PID: 17289
I20260928 10:53:05.036774 6030 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.227.148:45505
Sep 28, 2026 10:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 10:53:07.562666 6030 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.123s user 0.142s sys 0.414s
I20260928 10:53:07.562948 6030 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 10:53:07.563164 6030 mini_ranger_kms.cc:401] 127.5.227.148:45505/kms/v1/keys
Sep 28 10:53:07 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.227.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 10:53:08 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.227.148@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
Sep 28 10:53:08 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592787, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.227.148@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
Sep 28 10:53:11 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:53:11.725765 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:43841
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:43841
--ranger_config_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:45505/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:53:12.174371 17339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:53:12.174700 17339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:53:12.174853 17339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:53:12.183552 17339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:53:12.183704 17339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:53:12.183828 17339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:53:12.183943 17339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:53:12.184006 17339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:53:12.196666 17339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.227.148:45505/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:43841
--ranger_config_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:43841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:53:12.198576 17339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:53:12.200817 17339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:53:12.219499 17344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:53:12.219482 17345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:53:12.219545 17347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:53:12.220688 17339 server_base.cc:1034] running on GCE node
I20260928 10:53:12.221877 17339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:53:12.224079 17339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:53:12.225440 17339 hybrid_clock.cc:648] HybridClock initialized: now 1790592792225369 us; error 64 us; skew 500 ppm
I20260928 10:53:12.226089 17339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:53:12 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592792, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:53:12.231323 17339 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:53:12.233487 17339 webserver.cc:460] Webserver started at http://127.5.227.190:43251/ using document root <none> and password file <none>
I20260928 10:53:12.234297 17339 fs_manager.cc:359] Metadata directory not provided
I20260928 10:53:12.234457 17339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:53:12.235008 17339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:53:12 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592792, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:53:12.303527 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "12f6b2d996634852942fc652430138c4"
format_stamp: "Formatted at 2026-09-28 10:53:12 on dist-test-slave-8w8g"
server_key: "62bc251481fe27be17e064e42e3a4c32"
server_key_iv: "d62bab3fc15226beffb6e5a2d8bf0561"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:12.304811 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "12f6b2d996634852942fc652430138c4"
format_stamp: "Formatted at 2026-09-28 10:53:12 on dist-test-slave-8w8g"
server_key: "62bc251481fe27be17e064e42e3a4c32"
server_key_iv: "d62bab3fc15226beffb6e5a2d8bf0561"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:12.312748 17339 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20260928 10:53:12.336705 17354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:12.338707 17339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:53:12.338948 17339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "12f6b2d996634852942fc652430138c4"
format_stamp: "Formatted at 2026-09-28 10:53:12 on dist-test-slave-8w8g"
server_key: "62bc251481fe27be17e064e42e3a4c32"
server_key_iv: "d62bab3fc15226beffb6e5a2d8bf0561"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:12.339229 17339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:53:12.379194 17339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:53:12.384946 17339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:53:12.385345 17339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:53:12.411638 17339 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:43841
I20260928 10:53:12.411648 17405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:43841 every 8 connection(s)
I20260928 10:53:12.413443 17339 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:53:12.417855 17406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:53:12.418895 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 17339
I20260928 10:53:12.419123 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:53:12.420224 6030 external_mini_cluster.cc:1428] Setting key 48960f3eabd40d943dca4ece04106618
I20260928 10:53:12.435159 17406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4: Bootstrap starting.
Sep 28 10:53:12 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:53:12.440127 17406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4: Neither blocks nor log segments found. Creating new log.
I20260928 10:53:12.441838 17406 log.cc:824] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:53:12.445952 17406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4: No bootstrap required, opened a new log
I20260928 10:53:12.453531 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:12.423244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.5.227.190:43841)
0928 10:53:12.424422 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:12.424440 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:12.424497 (+ 57us) server_negotiation.cc:407] Connection header received
0928 10:53:12.426522 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:12.426558 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:12.427240 (+ 682us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:12.427828 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:12.429963 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:12.432231 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:12.434153 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:12.435488 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:12.435788 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:12.441636 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:12.441652 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:12.441677 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:12.441736 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:12.446896 (+ 5160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:12.448089 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:12.448099 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:12.448110 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:12.448376 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:12.449308 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:12.449314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:12.449317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:12.450241 (+ 924us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:12.450578 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:12.451270 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 10:53:12.451833 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":611,"threads_started":1}
I20260928 10:53:12.457096 17406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6b2d996634852942fc652430138c4" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43841 } }
I20260928 10:53:12.457659 17406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:53:12.457828 17406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f6b2d996634852942fc652430138c4, State: Initialized, Role: FOLLOWER
I20260928 10:53:12.458503 17406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6b2d996634852942fc652430138c4" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43841 } }
I20260928 10:53:12.458806 17406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:53:12.458966 17406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:53:12.459189 17406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:53:12.462059 17406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6b2d996634852942fc652430138c4" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43841 } }
I20260928 10:53:12.462687 17406 leader_election.cc:302] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f6b2d996634852942fc652430138c4; no voters:
I20260928 10:53:12.463603 17406 leader_election.cc:288] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:53:12.463754 17411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:53:12.466491 17411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [term 1 LEADER]: Becoming Leader. State: Replica: 12f6b2d996634852942fc652430138c4, State: Running, Role: LEADER
I20260928 10:53:12.468015 17411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6b2d996634852942fc652430138c4" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43841 } }
I20260928 10:53:12.468937 17406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:53:12.473117 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f6b2d996634852942fc652430138c4. Latest consensus state: current_term: 1 leader_uuid: "12f6b2d996634852942fc652430138c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6b2d996634852942fc652430138c4" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43841 } } }
I20260928 10:53:12.473721 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:53:12.474545 17412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f6b2d996634852942fc652430138c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6b2d996634852942fc652430138c4" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43841 } } }
I20260928 10:53:12.474922 17412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:53:12.485602 17416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:53:12.494130 17416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:53:12.495066 17406 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 10:53:12.515865 17416 catalog_manager.cc:1347] Generated new cluster ID: 8ae1f93c9249438c829cb9e1b742dda9
I20260928 10:53:12.516038 17416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:53:12.533949 17416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:53:12.536175 17416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:53:12.548555 17416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4: Generated new TSK 0
I20260928 10:53:12.549816 17416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:53:13.490103 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:45505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:43841
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:53:13 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 10:53:14.019992 17451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:53:14.020532 17451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:53:14.020738 17451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:53:14.035147 17451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:53:14.035347 17451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:53:14.035542 17451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:53:14.054106 17451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.227.148:45505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:53:14.055987 17451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:53:14.058296 17451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:53:14.102141 17461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:53:14.105111 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:53:14.116601 17451 server_base.cc:1034] running on GCE node
W20260928 10:53:14.116420 17464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:53:14.118113 17451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:53:14.121539 17451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:53:14.125236 17451 hybrid_clock.cc:648] HybridClock initialized: now 1790592794125116 us; error 95 us; skew 500 ppm
I20260928 10:53:14.125744 17451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:53:14 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592794, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:53:14.130507 17451 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:53:14.140233 17451 webserver.cc:460] Webserver started at http://127.5.227.129:45237/ using document root <none> and password file <none>
I20260928 10:53:14.141494 17451 fs_manager.cc:359] Metadata directory not provided
I20260928 10:53:14.141640 17451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:53:14.142081 17451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:53:14 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592794, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:53:14.170141 17451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "f2a97c6047654b13b380b1725ad691e0"
format_stamp: "Formatted at 2026-09-28 10:53:14 on dist-test-slave-8w8g"
server_key: "f605c1e253a495d0c6e5edd81b3a21ec"
server_key_iv: "a9dd840af6572308c8e7521d3fa2c610"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:14.172056 17451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "f2a97c6047654b13b380b1725ad691e0"
format_stamp: "Formatted at 2026-09-28 10:53:14 on dist-test-slave-8w8g"
server_key: "f605c1e253a495d0c6e5edd81b3a21ec"
server_key_iv: "a9dd840af6572308c8e7521d3fa2c610"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:14.209147 17451 fs_manager.cc:691] Time spent creating directory manager: real 0.036s user 0.022s sys 0.010s
I20260928 10:53:14.252993 17471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:14.258508 17451 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.004s sys 0.008s
I20260928 10:53:14.258780 17451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "f2a97c6047654b13b380b1725ad691e0"
format_stamp: "Formatted at 2026-09-28 10:53:14 on dist-test-slave-8w8g"
server_key: "f605c1e253a495d0c6e5edd81b3a21ec"
server_key_iv: "a9dd840af6572308c8e7521d3fa2c610"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:14.259050 17451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:53:14.284183 17451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:53:14.291951 17451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:53:14.292464 17451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:53:14.293821 17451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:53:14.300602 17451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:53:14.300853 17451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 10:53:14.301096 17451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:53:14.301199 17451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:53:14 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:53:14.465250 17451 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:38453
I20260928 10:53:14.465257 17583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:38453 every 8 connection(s)
I20260928 10:53:14.467716 17451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:53:14.471621 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 17451
I20260928 10:53:14.472014 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:53:14.473183 6030 external_mini_cluster.cc:1428] Setting key dc2febc8798ebffaeccfc7f231100bc6
Sep 28 10:53:14 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592794, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
I20260928 10:53:14.537431 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:14.479251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:58645 (local address 127.5.227.190:43841)
0928 10:53:14.505330 (+ 26079us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:14.505341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:14.510909 (+ 5568us) server_negotiation.cc:407] Connection header received
0928 10:53:14.510995 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:14.511003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:14.511136 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:14.511262 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:14.516059 (+ 4797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:14.517593 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:14.520528 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:14.521128 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:14.521239 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:14.526803 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:14.526811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:14.526816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:14.526866 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:14.531903 (+ 5037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:14.533500 (+ 1597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:14.533508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:14.533512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:14.533717 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:14.534394 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:14.534400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:14.534404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:14.534802 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:14.534967 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:14.536625 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0928 10:53:14.536904 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25895,"thread_start_us":26077,"threads_started":1}
I20260928 10:53:14.540469 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:14.478536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:43841 (local address 127.5.227.129:58645)
0928 10:53:14.506311 (+ 27775us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:14.506377 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:14.510284 (+ 3907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:14.512987 (+ 2703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:14.513008 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:14.514104 (+ 1096us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:53:14.515612 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:14.515680 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:14.518048 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:14.518054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:14.520269 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:14.520306 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:14.521663 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:14.521672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:14.522027 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:14.522943 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:53:14.522986 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:53:14.526395 (+ 3409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:53:14.532184 (+ 5789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:53:14.532196 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:53:14.532218 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:53:14.533283 (+ 1065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:53:14.533907 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:53:14.533912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:53:14.533916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:53:14.534238 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:53:14.535077 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:53:14.535088 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:53:14.536087 (+ 999us) client_negotiation.cc:769] Sending connection context
0928 10:53:14.537961 (+ 1874us) client_negotiation.cc:240] Negotiation successful
0928 10:53:14.538635 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27128,"thread_start_us":32785,"threads_started":1}
I20260928 10:53:14.543453 17584 heartbeater.cc:347] Connected to a master server at 127.5.227.190:43841
I20260928 10:53:14.544147 17584 heartbeater.cc:464] Registering TS with master...
I20260928 10:53:14.545157 17584 heartbeater.cc:511] Master 127.5.227.190:43841 requested a full tablet report, sending...
I20260928 10:53:14.549093 17371 ts_manager.cc:194] Registered new tserver with Master: f2a97c6047654b13b380b1725ad691e0 (127.5.227.129:38453)
I20260928 10:53:14.551713 17371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:58645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
W20260928 10:53:14.589874 6030 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.227.130: real 0.112s user 0.002s sys 0.019s
I20260928 10:53:14.591331 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:45505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:43841
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:53:14 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592793, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
W20260928 10:53:15.185632 17590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:53:15.185969 17590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:53:15.186419 17590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:53:15.194971 17590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:53:15.195125 17590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:53:15.195334 17590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:53:15.208786 17590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.227.148:45505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:53:15.210271 17590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:53:15.211849 17590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:53:15.231511 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:53:15.231496 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:53:15.233249 17590 server_base.cc:1034] running on GCE node
W20260928 10:53:15.233477 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:53:15.234706 17590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:53:15.235971 17590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:53:15.237288 17590 hybrid_clock.cc:648] HybridClock initialized: now 1790592795237198 us; error 67 us; skew 500 ppm
I20260928 10:53:15.237722 17590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:53:15 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592795, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:53:15.247277 17590 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:53:15.249509 17590 webserver.cc:460] Webserver started at http://127.5.227.130:39079/ using document root <none> and password file <none>
I20260928 10:53:15.250422 17590 fs_manager.cc:359] Metadata directory not provided
I20260928 10:53:15.250598 17590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:53:15.250962 17590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:53:15 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592795, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:53:15.272852 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "1f70093d72834599a40ee3c32ea3a825"
format_stamp: "Formatted at 2026-09-28 10:53:15 on dist-test-slave-8w8g"
server_key: "13662218a10c52f4f638dd9ebca2d440"
server_key_iv: "efe3372d82354d2224c1371b56b30591"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:15.274696 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "1f70093d72834599a40ee3c32ea3a825"
format_stamp: "Formatted at 2026-09-28 10:53:15 on dist-test-slave-8w8g"
server_key: "13662218a10c52f4f638dd9ebca2d440"
server_key_iv: "efe3372d82354d2224c1371b56b30591"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:15.283033 17590 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260928 10:53:15.304888 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:15.307214 17590 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:53:15.307488 17590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "1f70093d72834599a40ee3c32ea3a825"
format_stamp: "Formatted at 2026-09-28 10:53:15 on dist-test-slave-8w8g"
server_key: "13662218a10c52f4f638dd9ebca2d440"
server_key_iv: "efe3372d82354d2224c1371b56b30591"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:15.307787 17590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:53:15.335908 17590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:53:15.341562 17590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:53:15.341934 17590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:53:15.343218 17590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:53:15.345067 17590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:53:15.345188 17590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:15.345348 17590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:53:15.345422 17590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:15.390950 17590 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:33517
I20260928 10:53:15.390961 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:33517 every 8 connection(s)
I20260928 10:53:15.392942 17590 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:53:15.393380 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 17590
I20260928 10:53:15.393603 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:53:15.394555 6030 external_mini_cluster.cc:1428] Setting key 394c08328b2678dedc12f7b49688fe6a
Sep 28 10:53:15 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592795, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:53:15.429606 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:15.399036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:49137 (local address 127.5.227.190:43841)
0928 10:53:15.404232 (+ 5196us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:15.404239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:15.404264 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:53:15.404333 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:15.404338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:15.404423 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:15.404705 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:15.407711 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:15.409276 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:15.411850 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:15.412654 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:15.412825 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:15.419028 (+ 6203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:15.419035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:15.419040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:15.419093 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:15.425077 (+ 5984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:15.426815 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:15.426821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:15.426827 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:15.427038 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:15.427580 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:15.427589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:15.427594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:15.427990 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:15.428148 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:15.428955 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 10:53:15.429210 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5047,"thread_start_us":4902,"threads_started":1}
I20260928 10:53:15.431190 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:15.397978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:43841 (local address 127.5.227.130:49137)
0928 10:53:15.399198 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:15.399249 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:15.401638 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:15.405080 (+ 3442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:15.405100 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:15.406199 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:53:15.407383 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:15.407434 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:15.409546 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:15.409553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:15.411544 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:15.411577 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:15.413039 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:15.413047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:15.413439 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:15.414861 (+ 1422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:53:15.414894 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:53:15.418341 (+ 3447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:53:15.425398 (+ 7057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:53:15.425410 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:53:15.425432 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:53:15.426496 (+ 1064us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:53:15.427230 (+ 734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:53:15.427235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:53:15.427237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:53:15.427448 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:53:15.428147 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:53:15.428158 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:53:15.428546 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 10:53:15.428852 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 10:53:15.429460 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":996,"threads_started":1}
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:53:15.433022 17727 heartbeater.cc:347] Connected to a master server at 127.5.227.190:43841
I20260928 10:53:15.433387 17727 heartbeater.cc:464] Registering TS with master...
I20260928 10:53:15.434445 17727 heartbeater.cc:511] Master 127.5.227.190:43841 requested a full tablet report, sending...
I20260928 10:53:15.436350 17371 ts_manager.cc:194] Registered new tserver with Master: 1f70093d72834599a40ee3c32ea3a825 (127.5.227.130:33517)
I20260928 10:53:15.437189 17371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:49137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:53:15.488348 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:45505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:43841
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:53:15.556579 17584 heartbeater.cc:503] Master 127.5.227.190:43841 was elected leader, sending a full tablet report...
W20260928 10:53:15.945233 17733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:53:15.945537 17733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:53:15.945693 17733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:53:15.953491 17733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:53:15.953625 17733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:53:15.953823 17733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:53:15.965642 17733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.227.148:45505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:53:15.967067 17733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:53:15.968497 17733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:53:15.983819 17738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:53:15.985817 17733 server_base.cc:1034] running on GCE node
W20260928 10:53:15.987480 17741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:53:15.987874 17739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:53:15.988888 17733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:53:15.990242 17733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:53:15.991616 17733 hybrid_clock.cc:648] HybridClock initialized: now 1790592795991541 us; error 69 us; skew 500 ppm
I20260928 10:53:15.992187 17733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:53:15 dist-test-slave-8w8g krb5kdc[16895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592795, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:53:15.996225 17733 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:53:15.998168 17733 webserver.cc:460] Webserver started at http://127.5.227.131:39523/ using document root <none> and password file <none>
I20260928 10:53:15.998955 17733 fs_manager.cc:359] Metadata directory not provided
I20260928 10:53:15.999122 17733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:53:15.999521 17733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:53:15 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592795, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:53:16.015728 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "ebddb0dedd56444899bd4d66b5759ab0"
format_stamp: "Formatted at 2026-09-28 10:53:16 on dist-test-slave-8w8g"
server_key: "6118e673571945ae6cc404894acfea5f"
server_key_iv: "2ef21b2f3776652129403a919fbc82fd"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:16.016857 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "ebddb0dedd56444899bd4d66b5759ab0"
format_stamp: "Formatted at 2026-09-28 10:53:16 on dist-test-slave-8w8g"
server_key: "6118e673571945ae6cc404894acfea5f"
server_key_iv: "2ef21b2f3776652129403a919fbc82fd"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:16.023505 17733 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 10:53:16.045495 17748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:16.047617 17733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:53:16.047914 17733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "ebddb0dedd56444899bd4d66b5759ab0"
format_stamp: "Formatted at 2026-09-28 10:53:16 on dist-test-slave-8w8g"
server_key: "6118e673571945ae6cc404894acfea5f"
server_key_iv: "2ef21b2f3776652129403a919fbc82fd"
server_key_version: "kuduclusterkey@0"
I20260928 10:53:16.048136 17733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:53:16.068116 17733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:53:16.075321 17733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:53:16.076488 17733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:53:16.078111 17733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:53:16.079977 17733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:53:16.080534 17733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:16.080709 17733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:53:16.080852 17733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:53:16.134374 17733 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:38225
I20260928 10:53:16.134431 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:38225 every 8 connection(s)
I20260928 10:53:16.135915 17733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:53:16.143963 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 17733
I20260928 10:53:16.144160 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:53:16.144739 6030 external_mini_cluster.cc:1428] Setting key 4b32cc597d336f8446ee2ea360e5c075
Sep 28 10:53:16 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592795, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:53:16.171799 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.140545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:41593 (local address 127.5.227.190:43841)
0928 10:53:16.141208 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.141217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.142776 (+ 1559us) server_negotiation.cc:407] Connection header received
0928 10:53:16.145800 (+ 3024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.145806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.145909 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.146041 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:16.149136 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.150915 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.154856 (+ 3941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.155659 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.155849 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.162399 (+ 6550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:16.162407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:16.162413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:16.162470 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:16.166262 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.167477 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.167482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.167485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.167897 (+ 412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.168847 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.168852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.168854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.169493 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:16.169870 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.170930 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.171324 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":347,"threads_started":1}
I20260928 10:53:16.173018 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.141301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:43841 (local address 127.5.227.131:41593)
0928 10:53:16.142374 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:16.142540 (+ 166us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:16.145408 (+ 2868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:16.146375 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:16.146390 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:16.147447 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:53:16.148875 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.148915 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.151246 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.151258 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.154540 (+ 3282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.154566 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.156287 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.156296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.156917 (+ 621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.158824 (+ 1907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:53:16.158863 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:53:16.162078 (+ 3215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:53:16.166489 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:53:16.166502 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:53:16.166525 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:53:16.167282 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:53:16.168045 (+ 763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:53:16.168050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:53:16.168054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:53:16.168624 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:53:16.169731 (+ 1107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:53:16.169742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:53:16.170358 (+ 616us) client_negotiation.cc:769] Sending connection context
0928 10:53:16.170735 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 10:53:16.171357 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":371,"threads_started":1}
I20260928 10:53:16.175280 17861 heartbeater.cc:347] Connected to a master server at 127.5.227.190:43841
I20260928 10:53:16.175755 17861 heartbeater.cc:464] Registering TS with master...
I20260928 10:53:16.177846 17861 heartbeater.cc:511] Master 127.5.227.190:43841 requested a full tablet report, sending...
I20260928 10:53:16.180444 17370 ts_manager.cc:194] Registered new tserver with Master: ebddb0dedd56444899bd4d66b5759ab0 (127.5.227.131:38225)
I20260928 10:53:16.182066 17370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:41593
I20260928 10:53:16.191565 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:53:16.219009 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.197613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.5.227.190:43841)
0928 10:53:16.197989 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.197998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.198953 (+ 955us) server_negotiation.cc:407] Connection header received
0928 10:53:16.199987 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.200002 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.200124 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.200246 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:16.203544 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.204580 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.207072 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.207517 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.207639 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.211791 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:16.211798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:16.211801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:16.211848 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:16.215047 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.216295 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.216300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.216303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.216468 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.217050 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.217078 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.217080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.217406 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:16.217614 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.218481 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.218767 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260928 10:53:16.226301 17370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:53:16.237267 17370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:53:16.272563 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.255462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:38453 (local address 127.0.0.1:39142)
0928 10:53:16.256612 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:16.256804 (+ 192us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:16.257081 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:16.263090 (+ 6009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:16.263095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:16.263118 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:53:16.263545 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.263558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.265946 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.265954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.268821 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.268836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.271001 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.271009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.271368 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.271453 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 10:53:16.271574 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 10:53:16.272056 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":3,"thread_start_us":511,"threads_started":1}
I20260928 10:53:16.273247 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.256837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.5.227.130:33517)
0928 10:53:16.259235 (+ 2398us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.259249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.259285 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:53:16.259461 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.259482 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.259961 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.260444 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:53:16.262021 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.265279 (+ 3258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.269310 (+ 4031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.271301 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.271463 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.271651 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.272440 (+ 789us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.272653 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2177,"thread_start_us":2031,"threads_started":1}
I20260928 10:53:16.274431 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.255658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.5.227.129:38453)
0928 10:53:16.262348 (+ 6690us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.262358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.262387 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:53:16.262487 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.262497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.262807 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.264052 (+ 1245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:53:16.264416 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.265794 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.269029 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.270699 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.270895 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.273295 (+ 2400us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.273410 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.273604 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6467,"thread_start_us":6139,"threads_started":1}
I20260928 10:53:16.274963 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.255979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:33517 (local address 127.0.0.1:50822)
0928 10:53:16.258344 (+ 2365us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:16.258401 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:16.258589 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:16.260431 (+ 1842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:16.260446 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:16.260503 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:53:16.261480 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.261503 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.265487 (+ 3984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.265493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.269078 (+ 3585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.269093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.271963 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.271971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.272138 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.272179 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 10:53:16.273477 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0928 10:53:16.273652 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2200,"mutex_wait_us":78,"thread_start_us":5262,"threads_started":1}
I20260928 10:53:16.275079 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.258528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.5.227.131:38225)
0928 10:53:16.259277 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.259287 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.262422 (+ 3135us) server_negotiation.cc:407] Connection header received
0928 10:53:16.262919 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.262926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.263246 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.263533 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:53:16.264967 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.266979 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.270070 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.272205 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.272362 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.272607 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.274384 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.274615 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":585,"threads_started":1}
I20260928 10:53:16.275213 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.257443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:38225 (local address 127.0.0.1:40544)
0928 10:53:16.261918 (+ 4475us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:16.261952 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:16.262123 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:16.264164 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:16.264168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:16.264203 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:53:16.264625 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.264639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.267214 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.267219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.269842 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.269856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.273219 (+ 3363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.273228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.273900 (+ 672us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.273956 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 10:53:16.274053 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 10:53:16.274242 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4027,"mutex_wait_us":76,"thread_start_us":690,"threads_started":1}
I20260928 10:53:16.279979 17519 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae680569e8448fa823288d2b9d66d0b (DEFAULT_TABLE table=test-table [id=37ae9ebe5ed14f839341a9445054e82b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:53:16.279986 17796 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae680569e8448fa823288d2b9d66d0b (DEFAULT_TABLE table=test-table [id=37ae9ebe5ed14f839341a9445054e82b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:53:16.281517 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae680569e8448fa823288d2b9d66d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:53:16.281895 17796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae680569e8448fa823288d2b9d66d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:53:16.283291 17662 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae680569e8448fa823288d2b9d66d0b (DEFAULT_TABLE table=test-table [id=37ae9ebe5ed14f839341a9445054e82b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:53:16.285028 17662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae680569e8448fa823288d2b9d66d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:53:16.297488 17880 tablet_bootstrap.cc:492] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Bootstrap starting.
I20260928 10:53:16.301505 17881 tablet_bootstrap.cc:492] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Bootstrap starting.
I20260928 10:53:16.305497 17882 tablet_bootstrap.cc:492] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Bootstrap starting.
I20260928 10:53:16.306145 17880 tablet_bootstrap.cc:654] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Neither blocks nor log segments found. Creating new log.
I20260928 10:53:16.307381 17881 tablet_bootstrap.cc:654] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Neither blocks nor log segments found. Creating new log.
I20260928 10:53:16.307968 17880 log.cc:824] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:53:16.308980 17881 log.cc:824] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:53:16.310480 17882 tablet_bootstrap.cc:654] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Neither blocks nor log segments found. Creating new log.
I20260928 10:53:16.311790 17882 log.cc:824] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Log is configured to *not* fsync() on all Append() calls
I20260928 10:53:16.313264 17880 tablet_bootstrap.cc:492] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: No bootstrap required, opened a new log
I20260928 10:53:16.313863 17880 ts_tablet_manager.cc:1397] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20260928 10:53:16.314416 17881 tablet_bootstrap.cc:492] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: No bootstrap required, opened a new log
I20260928 10:53:16.314774 17881 ts_tablet_manager.cc:1397] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260928 10:53:16.318885 17882 tablet_bootstrap.cc:492] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: No bootstrap required, opened a new log
I20260928 10:53:16.319274 17882 ts_tablet_manager.cc:1397] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20260928 10:53:16.325048 17880 raft_consensus.cc:348] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.325520 17880 raft_consensus.cc:374] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:53:16.325634 17880 raft_consensus.cc:729] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a97c6047654b13b380b1725ad691e0, State: Initialized, Role: FOLLOWER
I20260928 10:53:16.326407 17880 consensus_queue.cc:260] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.329255 17881 raft_consensus.cc:348] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.329798 17881 raft_consensus.cc:374] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:53:16.329892 17881 raft_consensus.cc:729] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebddb0dedd56444899bd4d66b5759ab0, State: Initialized, Role: FOLLOWER
I20260928 10:53:16.330768 17881 consensus_queue.cc:260] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.333297 17880 ts_tablet_manager.cc:1428] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Time spent starting tablet: real 0.019s user 0.010s sys 0.010s
I20260928 10:53:16.333845 17882 raft_consensus.cc:348] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.334401 17882 raft_consensus.cc:374] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:53:16.334515 17882 raft_consensus.cc:729] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f70093d72834599a40ee3c32ea3a825, State: Initialized, Role: FOLLOWER
I20260928 10:53:16.334749 17861 heartbeater.cc:503] Master 127.5.227.190:43841 was elected leader, sending a full tablet report...
I20260928 10:53:16.335688 17881 ts_tablet_manager.cc:1428] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260928 10:53:16.335569 17882 consensus_queue.cc:260] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.338183 17727 heartbeater.cc:503] Master 127.5.227.190:43841 was elected leader, sending a full tablet report...
I20260928 10:53:16.338848 17882 ts_tablet_manager.cc:1428] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Time spent starting tablet: real 0.019s user 0.014s sys 0.007s
I20260928 10:53:16.342981 17886 raft_consensus.cc:482] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:53:16.343434 17886 raft_consensus.cc:504] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.347949 17886 leader_election.cc:288] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebddb0dedd56444899bd4d66b5759ab0 (127.5.227.131:38225), 1f70093d72834599a40ee3c32ea3a825 (127.5.227.130:33517)
I20260928 10:53:16.363868 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.346433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:48779 (local address 127.5.227.130:33517)
0928 10:53:16.346663 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.346669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.352902 (+ 6233us) server_negotiation.cc:407] Connection header received
0928 10:53:16.353176 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.353183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.353272 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.353361 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:53:16.354902 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.356114 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.359953 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.361706 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.361949 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.362030 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.363142 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.363305 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:53:16.363924 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.346472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:34215 (local address 127.5.227.131:38225)
0928 10:53:16.347133 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.347220 (+ 87us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.352920 (+ 5700us) server_negotiation.cc:407] Connection header received
0928 10:53:16.353009 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.353021 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.353132 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.353263 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:53:16.354904 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.356114 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.359946 (+ 3832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.361707 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.361967 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.362041 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.363134 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.363305 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
I20260928 10:53:16.363970 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.346454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:38225 (local address 127.5.227.129:34215)
0928 10:53:16.352707 (+ 6253us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:16.352724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:16.352884 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:16.353494 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:16.353499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:16.353536 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:53:16.354620 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.354647 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.356355 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.356361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.359453 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.359482 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.362267 (+ 2785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.362275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.362608 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.362849 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 10:53:16.363003 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 10:53:16.363332 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6114,"mutex_wait_us":496,"spinlock_wait_cycles":56576,"thread_start_us":5515,"threads_started":1}
I20260928 10:53:16.365257 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.346454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:33517 (local address 127.5.227.129:48779)
0928 10:53:16.352536 (+ 6082us) negotiation.cc:107] Waiting for socket to connect
0928 10:53:16.352564 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:53:16.353012 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:53:16.353494 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:53:16.353499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:53:16.353538 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:53:16.354659 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.354670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.356355 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.356361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.359463 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:53:16.359482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.362266 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.362275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:53:16.362613 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.362865 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 10:53:16.363003 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 10:53:16.363324 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5811,"mutex_wait_us":546,"spinlock_wait_cycles":23936,"thread_start_us":5869,"threads_started":1}
I20260928 10:53:16.365566 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae680569e8448fa823288d2b9d66d0b" candidate_uuid: "f2a97c6047654b13b380b1725ad691e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebddb0dedd56444899bd4d66b5759ab0" is_pre_election: true
I20260928 10:53:16.366611 17816 raft_consensus.cc:2454] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a97c6047654b13b380b1725ad691e0 in term 0.
I20260928 10:53:16.366676 17682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae680569e8448fa823288d2b9d66d0b" candidate_uuid: "f2a97c6047654b13b380b1725ad691e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f70093d72834599a40ee3c32ea3a825" is_pre_election: true
I20260928 10:53:16.367306 17682 raft_consensus.cc:2454] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a97c6047654b13b380b1725ad691e0 in term 0.
I20260928 10:53:16.367640 17474 leader_election.cc:302] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebddb0dedd56444899bd4d66b5759ab0, f2a97c6047654b13b380b1725ad691e0; no voters:
I20260928 10:53:16.368305 17886 raft_consensus.cc:2781] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:53:16.368474 17886 raft_consensus.cc:482] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:53:16.368587 17886 raft_consensus.cc:3037] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:53:16.371542 17886 raft_consensus.cc:504] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
I20260928 10:53:16.372413 17886 leader_election.cc:288] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [CANDIDATE]: Term 1 election: Requested vote from peers ebddb0dedd56444899bd4d66b5759ab0 (127.5.227.131:38225), 1f70093d72834599a40ee3c32ea3a825 (127.5.227.130:33517)
I20260928 10:53:16.373080 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae680569e8448fa823288d2b9d66d0b" candidate_uuid: "f2a97c6047654b13b380b1725ad691e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebddb0dedd56444899bd4d66b5759ab0"
I20260928 10:53:16.373170 17682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae680569e8448fa823288d2b9d66d0b" candidate_uuid: "f2a97c6047654b13b380b1725ad691e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f70093d72834599a40ee3c32ea3a825"
I20260928 10:53:16.373389 17816 raft_consensus.cc:3037] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:53:16.373395 17682 raft_consensus.cc:3037] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:53:16.377864 17816 raft_consensus.cc:2454] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a97c6047654b13b380b1725ad691e0 in term 1.
I20260928 10:53:16.377864 17682 raft_consensus.cc:2454] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a97c6047654b13b380b1725ad691e0 in term 1.
I20260928 10:53:16.378693 17472 leader_election.cc:302] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f70093d72834599a40ee3c32ea3a825, f2a97c6047654b13b380b1725ad691e0; no voters:
I20260928 10:53:16.379432 17886 raft_consensus.cc:2781] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:53:16.380262 17886 raft_consensus.cc:686] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 1 LEADER]: Becoming Leader. State: Replica: f2a97c6047654b13b380b1725ad691e0, State: Running, Role: LEADER
I20260928 10:53:16.381213 17886 consensus_queue.cc:237] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } }
W20260928 10:53:16.388413 17862 tablet.cc:2369] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:53:16.388875 17369 catalog_manager.cc:5495] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2a97c6047654b13b380b1725ad691e0 (127.5.227.129). New cstate: current_term: 1 leader_uuid: "f2a97c6047654b13b380b1725ad691e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a97c6047654b13b380b1725ad691e0" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 38453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 } health_report { overall_health: UNKNOWN } } }
W20260928 10:53:16.398212 17728 tablet.cc:2369] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:53:16.441496 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.433053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.5.227.129:38453)
0928 10:53:16.433289 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.433296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.433657 (+ 361us) server_negotiation.cc:407] Connection header received
0928 10:53:16.433893 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.433897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.434067 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.434235 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:53:16.435008 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.436470 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.438357 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.439202 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.439326 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.440085 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:53:16.440656 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:53:16.440801 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.441178 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.441278 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:53:16.452939 17682 raft_consensus.cc:1261] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 1 FOLLOWER]: Refusing update from remote peer f2a97c6047654b13b380b1725ad691e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:53:16.453024 17816 raft_consensus.cc:1261] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 1 FOLLOWER]: Refusing update from remote peer f2a97c6047654b13b380b1725ad691e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:53:16.454257 17891 consensus_queue.cc:1035] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f70093d72834599a40ee3c32ea3a825" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 33517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:53:16.454890 17886 consensus_queue.cc:1035] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebddb0dedd56444899bd4d66b5759ab0" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 38225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:53:16.477964 17894 mvcc.cc:205] Tried to move back new op lower bound from 7334268094257102848 to 7334268093982928896. Current Snapshot: MvccSnapshot[applied={T|T < 7334268094257102848}]
I20260928 10:53:16.483822 17895 mvcc.cc:205] Tried to move back new op lower bound from 7334268094257102848 to 7334268093982928896. Current Snapshot: MvccSnapshot[applied={T|T < 7334268094257102848}]
I20260928 10:53:16.503353 17370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:53:16.504137 17370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:53:16.509320 17370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12f6b2d996634852942fc652430138c4: Sending DeleteTablet for 3 replicas of tablet 1ae680569e8448fa823288d2b9d66d0b
I20260928 10:53:16.511286 17662 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae680569e8448fa823288d2b9d66d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:53:16 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:50822
I20260928 10:53:16.511286 17796 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae680569e8448fa823288d2b9d66d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:53:16 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:40544
I20260928 10:53:16.512084 17519 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae680569e8448fa823288d2b9d66d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:53:16 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:39142
I20260928 10:53:16.517118 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 17451
I20260928 10:53:16.522766 17905 tablet_replica.cc:330] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: stopping tablet replica
I20260928 10:53:16.522858 17904 tablet_replica.cc:330] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: stopping tablet replica
I20260928 10:53:16.523380 17904 raft_consensus.cc:2229] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:53:16.524073 17904 raft_consensus.cc:2258] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:53:16.524097 17905 raft_consensus.cc:2229] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:53:16.525492 17905 raft_consensus.cc:2258] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:53:16.526621 17904 ts_tablet_manager.cc:1905] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:53:16.529299 17905 ts_tablet_manager.cc:1905] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:53:16.535265 17904 ts_tablet_manager.cc:1918] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:53:16.536178 17903 tablet_replica.cc:330] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: stopping tablet replica
I20260928 10:53:16.536659 17904 log.cc:1196] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/1ae680569e8448fa823288d2b9d66d0b
I20260928 10:53:16.537122 17903 raft_consensus.cc:2229] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:53:16.537482 17904 ts_tablet_manager.cc:1939] T 1ae680569e8448fa823288d2b9d66d0b P 1f70093d72834599a40ee3c32ea3a825: Deleting consensus metadata
I20260928 10:53:16.538156 17903 raft_consensus.cc:2258] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:53:16.540478 17355 catalog_manager.cc:4841] TS 1f70093d72834599a40ee3c32ea3a825 (127.5.227.130:33517): tablet 1ae680569e8448fa823288d2b9d66d0b (table test-table [id=37ae9ebe5ed14f839341a9445054e82b]) successfully deleted
I20260928 10:53:16.541342 17903 ts_tablet_manager.cc:1905] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:53:16.545977 17905 ts_tablet_manager.cc:1918] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:53:16.546411 17905 log.cc:1196] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/1ae680569e8448fa823288d2b9d66d0b
I20260928 10:53:16.547326 17905 ts_tablet_manager.cc:1939] T 1ae680569e8448fa823288d2b9d66d0b P f2a97c6047654b13b380b1725ad691e0: Deleting consensus metadata
Sep 28 10:53:16 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:53:16.550575 17357 catalog_manager.cc:4841] TS f2a97c6047654b13b380b1725ad691e0 (127.5.227.129:38453): tablet 1ae680569e8448fa823288d2b9d66d0b (table test-table [id=37ae9ebe5ed14f839341a9445054e82b]) successfully deleted
I20260928 10:53:16.552229 17903 ts_tablet_manager.cc:1918] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:53:16.552434 17903 log.cc:1196] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/1ae680569e8448fa823288d2b9d66d0b
I20260928 10:53:16.553176 17903 ts_tablet_manager.cc:1939] T 1ae680569e8448fa823288d2b9d66d0b P ebddb0dedd56444899bd4d66b5759ab0: Deleting consensus metadata
I20260928 10:53:16.556450 17357 catalog_manager.cc:4841] TS ebddb0dedd56444899bd4d66b5759ab0 (127.5.227.131:38225): tablet 1ae680569e8448fa823288d2b9d66d0b (table test-table [id=37ae9ebe5ed14f839341a9445054e82b]) successfully deleted
I20260928 10:53:16.562269 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.519720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.5.227.129:38453)
0928 10:53:16.519959 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.519966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.521975 (+ 2009us) server_negotiation.cc:407] Connection header received
0928 10:53:16.524055 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.524062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.524171 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.524294 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:16.537630 (+ 13336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.539024 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.541537 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.542524 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.542709 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.551542 (+ 8833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:16.551563 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:16.551583 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:16.551646 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:16.556512 (+ 4866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.557911 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.557925 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.557930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.558634 (+ 704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.559668 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.559674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.559677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.560338 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:16.560732 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.561407 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.561697 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:53:16.563278 17579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:53:16.686367 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 17451
I20260928 10:53:16.708787 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 17590
Sep 28 10:53:16 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:53:16.731125 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.710060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.5.227.130:33517)
0928 10:53:16.710291 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.710298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.710330 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:53:16.710744 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.710756 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.710825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.710956 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:16.713067 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.714657 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.717197 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.717616 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.717786 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.723314 (+ 5528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:16.723332 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:16.723349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:16.723420 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:16.727334 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.728510 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.728518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.728521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.728686 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.729144 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.729148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.729150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.729600 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:16.729931 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.730288 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.730542 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:53:16.732081 17722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:53:16.839885 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 17590
I20260928 10:53:16.859395 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 17733
Sep 28 10:53:16 dist-test-slave-8w8g krb5kdc[16895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:53:16.880581 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:53:16.860557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.5.227.131:38225)
0928 10:53:16.860910 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 10:53:16.860915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:53:16.860934 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:53:16.860990 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:53:16.860993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:53:16.861066 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:53:16.861164 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:53:16.863918 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.864679 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.866219 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:53:16.866681 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:53:16.866811 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:53:16.872435 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:53:16.872455 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:53:16.872473 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:53:16.872526 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:53:16.876387 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.877472 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.877482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.877484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.877676 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:53:16.878310 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:53:16.878318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:53:16.878321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:53:16.879025 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:53:16.879294 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 10:53:16.879907 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 10:53:16.880126 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260928 10:53:16.881363 17856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:53:16.981022 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 17733
I20260928 10:53:16.999356 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 17339
I20260928 10:53:17.001165 17401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:53:17.125192 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 17339
I20260928 10:53:17.152518 6030 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 10:53:17.490021 6030 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 10:53:17.490263 6030 mini_ranger.cc:66] Stopping Ranger...
I20260928 10:53:17.837996 6030 mini_ranger.cc:68] Stopped Ranger
2026-09-28 10:53:17.838 UTC [16925] LOG: received smart shutdown request
2026-09-28 10:53:17.844 UTC [16925] LOG: background worker "logical replication launcher" (PID 16934) exited with exit code 1
2026-09-28 10:53:17.844 UTC [16929] LOG: shutting down
2026-09-28 10:53:18.238 UTC [16925] LOG: database system is shut down
2026-09-28T10:53:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (63876 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:53:18 dist-test-slave-8w8g krb5kdc[17919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:53:18 dist-test-slave-8w8g krb5kdc[17919](info): set up 2 sockets
Sep 28 10:53:18 dist-test-slave-8w8g krb5kdc[17919](info): commencing operation
krb5kdc: starting...
W20260928 10:53:19.701107 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:53:19 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:53:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/rangeradmin_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/rangeradmin_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/rangerlookup_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/rangerlookup_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
I20260928 10:53:20.117285 6030 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko9h9KU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/postgres -l logfile start
2026-09-28 10:53:24.729 UTC [17949] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 10:53:24.729 UTC [17949] LOG: listening on IPv4 address "127.5.227.148", port 52971
2026-09-28 10:53:24.736 UTC [17949] LOG: listening on Unix socket "/tmp/.s.PGSQL.52971"
2026-09-28 10:53:24.764 UTC [17952] LOG: database system was shut down at 2026-09-28 10:53:21 UTC
2026-09-28 10:53:24.770 UTC [17949] LOG: database system is ready to accept connections
I20260928 10:53:25.788609 6030 mini_postgres.cc:96] Postgres bound to 52971
2026-09-28 10:53:25.808 UTC [17960] FATAL: database "slave" does not exist
127.5.227.148:52971 - accepting connections
I20260928 10:53:25.808921 6030 mini_ranger.cc:161] Starting Ranger...
I20260928 10:53:25.839090 6030 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 10:53:26.768628 6030 mini_ranger.cc:87] Created ranger Postgres database
I20260928 10:53:26.768754 6030 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-admin
2026-09-28 10:53:27,125 [I] DB FLAVOR :POSTGRES
2026-09-28 10:53:27,126 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:53:27,126 [I] Checking connection..
2026-09-28 10:53:27,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 10:53:27,458 [I] Checking connection passed.
2026-09-28 10:53:27,458 [I] --------- Verifying version history table ---------
2026-09-28 10:53:27,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:53:27,770 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:53:27,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:53:28,042 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:53:28,042 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 10:53:28,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 10:53:28.371 UTC [18024] WARNING: there is no transaction in progress
2026-09-28 10:53:28,380 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 10:53:28,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:53:28,664 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 10:53:28,664 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:53:28,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 10:53:28,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 10:53:29,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8w8g.c.gcp-upstream.internal','N') ;"
2026-09-28 10:53:29,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:53:29,878 [I] Table x_portal_user does not exist in database ranger
2026-09-28 10:53:29,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:53:30,167 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 10:53:30,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:53:30,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:53:30,743 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 10:53:30,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 10:53:32.276 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.300 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.324 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.348 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.373 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.404 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.494 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.506 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.530 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.546 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.562 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.583 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.598 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.609 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.622 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:32.634 UTC [18132] WARNING: there is no transaction in progress
2026-09-28 10:53:33,749 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 10:53:33,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 10:53:34,039 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 10:53:34,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:53:34,320 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 10:53:34,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:53:34,598 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 10:53:34,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:53:34,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:53:35,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8w8g.c.gcp-upstream.internal';"
2026-09-28 10:53:35,504 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 10:53:35,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 10:53:35,834 [I] DB_PATCHES have already been applied
I20260928 10:53:35.842695 6030 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko9h9KU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 10:53:35.842854 6030 mini_ranger.cc:193] Using host: 127.5.227.148
I20260928 10:53:35.862424 6030 mini_ranger.cc:227] Ranger admin URL: http://127.5.227.148:52675
Sep 28, 2026 10:53:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:53:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:53:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:53:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:53:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:53:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:53:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 10:53:37.082480 6030 mini_ranger.cc:160] Time spent starting Ranger: real 11.274s user 0.000s sys 0.048s
Sep 28, 2026 10:54:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:54:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 10:54:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:54:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:54:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 10:54:02.781152 6030 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/rangerkms_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/rangerkms_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.227.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.227.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.227.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/HTTP_127.5.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 10:54:03 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592843, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 10:54:03.023406 6030 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 10:54:03.052681 6030 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 10:54:05.413291 6030 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 10:54:05.413439 6030 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms
I20260928 10:54:05.413475 6030 mini_ranger_kms.cc:227] Using postgres at 127.5.227.148:52971
2026-09-28 10:54:05,851 [I] DB FLAVOR :POSTGRES
2026-09-28 10:54:05,851 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:54:05,852 [I] Checking connection
2026-09-28 10:54:05,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 10:54:06,190 [I] connection success
2026-09-28 10:54:06,191 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 10:54:06,191 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 10:54:06,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 10:54:06,523 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 10:54:06,524 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:54:06,524 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 10:54:06,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.227.148:52971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 10:54:06,876 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 10:54:07.474865 6030 mini_ranger_kms.cc:310] Created kms service
I20260928 10:54:07.744462 6030 mini_ranger_kms.cc:326] Created kudu user
I20260928 10:54:07.812503 6030 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 10:54:07 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592843, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:08.326874 6030 mini_ranger_kms.cc:384] Added ranger policy
I20260928 10:54:08.327196 6030 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko9h9KU/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko9h9KU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 10:54:08.327288 6030 mini_ranger_kms.cc:240] Using host: 127.5.227.148
I20260928 10:54:08.348788 6030 mini_ranger_kms.cc:276] Ranger KMS PID: 18313
I20260928 10:54:08.348963 6030 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.227.148:56045
Sep 28, 2026 10:54:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:54:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:54:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:54:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:54:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:54:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 10:54:10.779171 6030 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.756s user 0.109s sys 0.341s
I20260928 10:54:10.779467 6030 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 10:54:10.779703 6030 mini_ranger_kms.cc:401] 127.5.227.148:56045/kms/v1/keys
Sep 28 10:54:10 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592850, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.227.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 10:54:11 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592850, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.227.148@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
Sep 28 10:54:11 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592850, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.227.148@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
Sep 28 10:54:14 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:54:14.850749 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39855
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39855
--ranger_config_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:15.259344 18363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:15.259701 18363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:15.259910 18363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:15.267661 18363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:15.267901 18363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:15.268009 18363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:15.268093 18363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:15.268378 18363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:15.280802 18363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.227.148:56045/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39855
--ranger_config_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:15.282311 18363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:15.284070 18363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 10:54:15.299595 18363 server_base.cc:1034] running on GCE node
W20260928 10:54:15.299821 18371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:15.300573 18368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:15.300793 18369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:15.301911 18363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:15.303930 18363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:15.305537 18363 hybrid_clock.cc:648] HybridClock initialized: now 1790592855305489 us; error 57 us; skew 500 ppm
I20260928 10:54:15.306299 18363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:15 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592855, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:15.310727 18363 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:54:15.312714 18363 webserver.cc:460] Webserver started at http://127.5.227.190:39771/ using document root <none> and password file <none>
I20260928 10:54:15.313488 18363 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:15.313730 18363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:15.314131 18363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:54:15 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592855, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:15.384086 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "db93f039c44d41ffa75bfe0e4b20304e"
format_stamp: "Formatted at 2026-09-28 10:54:15 on dist-test-slave-8w8g"
server_key: "797025e8a7ba4f9c6e11df9dcc3a52ad"
server_key_iv: "e24d2705c93bf60adfbe3a5b7d1d8606"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:15.385308 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "db93f039c44d41ffa75bfe0e4b20304e"
format_stamp: "Formatted at 2026-09-28 10:54:15 on dist-test-slave-8w8g"
server_key: "797025e8a7ba4f9c6e11df9dcc3a52ad"
server_key_iv: "e24d2705c93bf60adfbe3a5b7d1d8606"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:15.392952 18363 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260928 10:54:15.414592 18378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:15.416545 18363 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:54:15.416810 18363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "db93f039c44d41ffa75bfe0e4b20304e"
format_stamp: "Formatted at 2026-09-28 10:54:15 on dist-test-slave-8w8g"
server_key: "797025e8a7ba4f9c6e11df9dcc3a52ad"
server_key_iv: "e24d2705c93bf60adfbe3a5b7d1d8606"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:15.417075 18363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:15.440510 18363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:15.445552 18363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:15.445931 18363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:15.469444 18363 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:39855
I20260928 10:54:15.469480 18429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:39855 every 8 connection(s)
I20260928 10:54:15.470959 18363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:54:15.475088 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 18363
I20260928 10:54:15.475375 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:54:15.475884 18430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:15.476361 6030 external_mini_cluster.cc:1428] Setting key 535a0fc28d9065b6443bf5b7e6107887
I20260928 10:54:15.493072 18430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Bootstrap starting.
Sep 28 10:54:15 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:15.497112 18430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:15.498373 18430 log.cc:824] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:15.501916 18430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: No bootstrap required, opened a new log
I20260928 10:54:15.504874 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:15.478480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.5.227.190:39855)
0928 10:54:15.479380 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:15.479397 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:15.479444 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:54:15.481122 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:15.481158 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:15.481639 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:15.482033 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:15.484059 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:15.485964 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:15.488275 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:15.489312 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:15.489471 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:15.494744 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:15.494755 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:15.494771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:15.494826 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:15.498914 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:15.499899 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:15.499907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:15.499915 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:15.500063 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:15.500583 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:15.500588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:15.500591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:15.501690 (+ 1099us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:15.501974 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:15.503110 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0928 10:54:15.503619 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":447,"threads_started":1}
I20260928 10:54:15.512648 18430 raft_consensus.cc:348] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:15.513206 18430 raft_consensus.cc:374] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:15.513335 18430 raft_consensus.cc:729] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db93f039c44d41ffa75bfe0e4b20304e, State: Initialized, Role: FOLLOWER
I20260928 10:54:15.514129 18430 consensus_queue.cc:260] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:15.514475 18430 raft_consensus.cc:388] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:54:15.514657 18430 raft_consensus.cc:482] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:54:15.514886 18430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:15.517433 18430 raft_consensus.cc:504] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:15.518055 18430 leader_election.cc:302] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db93f039c44d41ffa75bfe0e4b20304e; no voters:
I20260928 10:54:15.518952 18430 leader_election.cc:288] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:54:15.519559 18435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:54:15.520332 18435 raft_consensus.cc:686] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 LEADER]: Becoming Leader. State: Replica: db93f039c44d41ffa75bfe0e4b20304e, State: Running, Role: LEADER
I20260928 10:54:15.521101 18435 consensus_queue.cc:237] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:15.521405 18430 sys_catalog.cc:564] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:15.525414 18437 sys_catalog.cc:455] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: SysCatalogTable state changed. Reason: New leader db93f039c44d41ffa75bfe0e4b20304e. Latest consensus state: current_term: 1 leader_uuid: "db93f039c44d41ffa75bfe0e4b20304e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } } }
I20260928 10:54:15.525990 18437 sys_catalog.cc:458] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:15.526831 18436 sys_catalog.cc:455] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db93f039c44d41ffa75bfe0e4b20304e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } } }
I20260928 10:54:15.527221 18436 sys_catalog.cc:458] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:15.532699 18443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:54:15.539131 18443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:54:15.540027 18430 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 10:54:15.559764 18443 catalog_manager.cc:1347] Generated new cluster ID: ba2ba8448d1c460b8f94f5145feadbab
I20260928 10:54:15.559926 18443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:54:15.578572 18443 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:54:15.581758 18443 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:54:15.593695 18443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Generated new TSK 0
I20260928 10:54:15.595060 18443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:54:16.463297 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39855
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:54:16 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 10:54:17.006286 18475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:17.006698 18475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:17.006916 18475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:17.016270 18475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:17.016425 18475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:17.016669 18475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:54:17.028000 18475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.227.148:56045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:17.029435 18475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:17.030869 18475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:17.052564 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:17.060104 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:17.061117 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:17.061450 18475 server_base.cc:1034] running on GCE node
I20260928 10:54:17.062371 18475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:17.069065 18475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:17.072641 18475 hybrid_clock.cc:648] HybridClock initialized: now 1790592857072482 us; error 132 us; skew 500 ppm
I20260928 10:54:17.073369 18475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:17 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:17.084916 18475 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:54:17.091060 18475 webserver.cc:460] Webserver started at http://127.5.227.129:43555/ using document root <none> and password file <none>
I20260928 10:54:17.092288 18475 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:17.092453 18475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:17.092837 18475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:54:17 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:17.116669 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "0db11ecf9ed745618f61abb9a4e1484f"
format_stamp: "Formatted at 2026-09-28 10:54:17 on dist-test-slave-8w8g"
server_key: "9441f5c424a2f0c52ea0f8b2afcf5bb4"
server_key_iv: "65a5e54f2e8dd85234b23d23a964d79c"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:17.118078 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "0db11ecf9ed745618f61abb9a4e1484f"
format_stamp: "Formatted at 2026-09-28 10:54:17 on dist-test-slave-8w8g"
server_key: "9441f5c424a2f0c52ea0f8b2afcf5bb4"
server_key_iv: "65a5e54f2e8dd85234b23d23a964d79c"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:17.134207 18475 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.009s sys 0.004s
I20260928 10:54:17.172900 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:17.175040 18475 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.004s
I20260928 10:54:17.175398 18475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "0db11ecf9ed745618f61abb9a4e1484f"
format_stamp: "Formatted at 2026-09-28 10:54:17 on dist-test-slave-8w8g"
server_key: "9441f5c424a2f0c52ea0f8b2afcf5bb4"
server_key_iv: "65a5e54f2e8dd85234b23d23a964d79c"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:17.175796 18475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:17.227823 18475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:17.235519 18475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:17.235989 18475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:17.243474 18475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:17.254982 18475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:17.255225 18475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:54:17.255456 18475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:17.255585 18475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:54:17 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:17.496793 18475 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:40711
I20260928 10:54:17.497246 18607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:40711 every 8 connection(s)
I20260928 10:54:17.498586 18475 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:54:17.503952 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 18475
I20260928 10:54:17.504252 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:54:17.505960 6030 external_mini_cluster.cc:1428] Setting key be6bdfee0e88daef048ad29885e5719e
Sep 28 10:54:17 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:17.541095 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:17.502579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39855 (local address 127.5.227.129:53885)
0928 10:54:17.512676 (+ 10097us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:17.512757 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:17.515417 (+ 2660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:17.517078 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:17.517091 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:17.517724 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:17.518612 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:17.518637 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:17.520251 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:17.520256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:17.522564 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:17.522593 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:17.524324 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:17.524331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:17.524652 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:17.525609 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:17.525651 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:17.529272 (+ 3621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:17.535049 (+ 5777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:17.535061 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:17.535085 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:17.535876 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:17.536497 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:17.536502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:17.536506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:17.536804 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:17.537596 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:17.537605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:17.538128 (+ 523us) client_negotiation.cc:769] Sending connection context
0928 10:54:17.538464 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 10:54:17.539085 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9677,"thread_start_us":9383,"threads_started":1}
I20260928 10:54:17.542560 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:17.503362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:53885 (local address 127.5.227.190:39855)
0928 10:54:17.516285 (+ 12923us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:17.516291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:17.516310 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:54:17.516375 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:17.516379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:17.516459 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:17.516576 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:17.518834 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:17.520003 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:17.522927 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:17.523807 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:17.523931 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:17.529725 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:17.529735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:17.529741 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:17.529808 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:17.534766 (+ 4958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:17.536070 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:17.536077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:17.536081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:17.536304 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:17.536952 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:17.536958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:17.536961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:17.537353 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:17.537501 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:17.541833 (+ 4332us) server_negotiation.cc:299] Negotiation successful
0928 10:54:17.542167 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12787,"thread_start_us":12646,"threads_started":1}
I20260928 10:54:17.545228 18608 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39855
I20260928 10:54:17.545568 18608 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:17.546456 18608 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
I20260928 10:54:17.549249 18395 ts_manager.cc:194] Registered new tserver with Master: 0db11ecf9ed745618f61abb9a4e1484f (127.5.227.129:40711)
I20260928 10:54:17.551286 18395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:53885
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:54:17.609676 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39855
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:54:17 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
W20260928 10:54:18.102597 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:18.102936 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:18.103109 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:18.111259 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:18.111500 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:18.112109 18614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:54:18.124794 18614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.227.148:56045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:18.127023 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:18.128834 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:18.145841 18628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:18.146193 18629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:18.146526 18631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:18.148397 18614 server_base.cc:1034] running on GCE node
I20260928 10:54:18.149361 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:18.150563 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:18.152005 18614 hybrid_clock.cc:648] HybridClock initialized: now 1790592858151906 us; error 87 us; skew 500 ppm
I20260928 10:54:18.152598 18614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:18 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592858, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:18.156539 18614 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:54:18.159001 18614 webserver.cc:460] Webserver started at http://127.5.227.130:38155/ using document root <none> and password file <none>
I20260928 10:54:18.159911 18614 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:18.160080 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:18.160420 18614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:54:18 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592858, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:18.180502 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "5e82fc306fca46a38edd20a38706a39f"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "1b80c13f90f4b7b6436bd3478fd8aea7"
server_key_iv: "b5c403c933d5042a8a4ab44ac176169b"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:18.181803 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "5e82fc306fca46a38edd20a38706a39f"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "1b80c13f90f4b7b6436bd3478fd8aea7"
server_key_iv: "b5c403c933d5042a8a4ab44ac176169b"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:18.191718 18614 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.003s
I20260928 10:54:18.212507 18638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:18.214395 18614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:54:18.214995 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "5e82fc306fca46a38edd20a38706a39f"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "1b80c13f90f4b7b6436bd3478fd8aea7"
server_key_iv: "b5c403c933d5042a8a4ab44ac176169b"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:18.215442 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:18.247960 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:18.253785 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:18.254158 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:18.255348 18614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:18.257000 18614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:18.257083 18614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:18.257211 18614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:18.257283 18614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:18.297792 18614 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:37829
I20260928 10:54:18.297854 18750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:37829 every 8 connection(s)
I20260928 10:54:18.299208 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:54:18.306576 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 18614
I20260928 10:54:18.306808 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:54:18.307338 6030 external_mini_cluster.cc:1428] Setting key 31aaeb15bade9d9c6941f96da5f2848d
Sep 28 10:54:18 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592858, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:18.330025 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:18.302370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:52583 (local address 127.5.227.190:39855)
0928 10:54:18.302927 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:18.302934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:18.304230 (+ 1296us) server_negotiation.cc:407] Connection header received
0928 10:54:18.306384 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:18.306401 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:18.306510 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:18.306668 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:18.308901 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:18.309945 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:18.313801 (+ 3856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:18.314287 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:18.314406 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:18.320120 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:18.320131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:18.320137 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:18.320204 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:18.324970 (+ 4766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:18.326453 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:18.326464 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:18.326470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:18.326696 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:18.327336 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:18.327345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:18.327353 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:18.327894 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:18.328079 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:18.329131 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0928 10:54:18.329443 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":261,"threads_started":1}
I20260928 10:54:18.330830 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:18.303015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39855 (local address 127.5.227.130:52583)
0928 10:54:18.304015 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:18.304077 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:18.305975 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:18.307233 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:18.307243 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:18.307877 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:18.308690 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:18.308712 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:18.310221 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:18.310228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:18.313535 (+ 3307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:18.313566 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:18.314592 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:18.314604 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:18.315034 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:18.316169 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:18.316211 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:18.319480 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:18.325605 (+ 6125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:18.325617 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:18.325634 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:18.326239 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:18.326861 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:18.326866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:18.326869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:18.327166 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:18.328110 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:18.328128 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:18.328649 (+ 521us) client_negotiation.cc:769] Sending connection context
0928 10:54:18.328932 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 10:54:18.329426 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":367,"threads_started":1}
I20260928 10:54:18.332965 18751 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39855
I20260928 10:54:18.333477 18751 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:18.334677 18751 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
I20260928 10:54:18.337169 18395 ts_manager.cc:194] Registered new tserver with Master: 5e82fc306fca46a38edd20a38706a39f (127.5.227.130:37829)
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:54:18.338302 18395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:52583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:54:18.389161 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39855
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:18.555853 18608 heartbeater.cc:503] Master 127.5.227.190:39855 was elected leader, sending a full tablet report...
W20260928 10:54:18.808528 18757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:18.808882 18757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:18.809072 18757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:18.819484 18757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:18.819667 18757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:18.820070 18757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:54:18.831725 18757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.227.148:56045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:39855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:18.833686 18757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:18.835101 18757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:18.852977 18765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:18.852952 18763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:18.852813 18762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:18.854342 18757 server_base.cc:1034] running on GCE node
I20260928 10:54:18.855365 18757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:18.857240 18757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:18.858706 18757 hybrid_clock.cc:648] HybridClock initialized: now 1790592858858634 us; error 67 us; skew 500 ppm
I20260928 10:54:18.859164 18757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:18 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592858, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:18.864073 18757 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:54:18.866498 18757 webserver.cc:460] Webserver started at http://127.5.227.131:42405/ using document root <none> and password file <none>
I20260928 10:54:18.867506 18757 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:18.867714 18757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:18.868193 18757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:54:18 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592858, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:18.886724 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "4fcdbe053a914bce8ca8b6e07a2a9058"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "cf08ce00f3c0f687d833747d6911e11d"
server_key_iv: "c7eb57311c307011bac8d56a6a3fb9d9"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:18.887841 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "4fcdbe053a914bce8ca8b6e07a2a9058"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "cf08ce00f3c0f687d833747d6911e11d"
server_key_iv: "c7eb57311c307011bac8d56a6a3fb9d9"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:18.894717 18757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 10:54:18.926518 18772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:18.929452 18757 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260928 10:54:18.929764 18757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "4fcdbe053a914bce8ca8b6e07a2a9058"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "cf08ce00f3c0f687d833747d6911e11d"
server_key_iv: "c7eb57311c307011bac8d56a6a3fb9d9"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:18.930130 18757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:18.966701 18757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:18.973507 18757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:18.973954 18757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:18.975955 18757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:18.978220 18757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:18.978364 18757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:18.978508 18757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:18.978579 18757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:19.027889 18757 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:42551
I20260928 10:54:19.027928 18884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:42551 every 8 connection(s)
I20260928 10:54:19.030071 18757 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:54:19.031195 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 18757
I20260928 10:54:19.031388 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:54:19.032069 6030 external_mini_cluster.cc:1428] Setting key e522e42ad9eadcadf2195e57433bcb37
Sep 28 10:54:19 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592858, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:19.069590 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:19.033898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:47411 (local address 127.5.227.190:39855)
0928 10:54:19.040340 (+ 6442us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:19.040349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:19.041209 (+ 860us) server_negotiation.cc:407] Connection header received
0928 10:54:19.044477 (+ 3268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:19.044487 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:19.044653 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:19.044813 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:19.046870 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.048520 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.051699 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.052733 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.052939 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:19.058583 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:19.058594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:19.058602 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:19.058674 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:19.065014 (+ 6340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.066142 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.066152 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.066158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.066392 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.067112 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.067121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.067126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.067665 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:19.067904 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:19.068734 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 10:54:19.069056 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6251,"thread_start_us":6080,"threads_started":1}
I20260928 10:54:19.070500 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:19.034560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39855 (local address 127.5.227.131:47411)
0928 10:54:19.040856 (+ 6296us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:19.040921 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:19.044040 (+ 3119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:19.045140 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:19.045154 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:19.045705 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:19.046594 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:19.046620 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.048828 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.048840 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:19.051452 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:19.051478 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.053070 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.053078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:19.053343 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:19.054419 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:19.054446 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:19.058058 (+ 3612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:19.065364 (+ 7306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:19.065380 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:19.065412 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:19.065964 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:19.066639 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:19.066643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:19.066646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:19.066949 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:19.067894 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:19.067901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:19.068304 (+ 403us) client_negotiation.cc:769] Sending connection context
0928 10:54:19.068577 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 10:54:19.069055 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5947,"thread_start_us":5650,"threads_started":1}
I20260928 10:54:19.073225 18885 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39855
I20260928 10:54:19.073609 18885 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:19.074417 18885 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
I20260928 10:54:19.076599 18395 ts_manager.cc:194] Registered new tserver with Master: 4fcdbe053a914bce8ca8b6e07a2a9058 (127.5.227.131:42551)
I20260928 10:54:19.078118 18395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:47411
I20260928 10:54:19.079942 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:54:19.084592 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 18475
Sep 28 10:54:19 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:54:19.110210 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:19.086159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.5.227.129:40711)
0928 10:54:19.086983 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:19.086993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:19.087021 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:54:19.087103 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:19.087109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:19.087336 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:19.087706 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:19.090683 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.092536 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.094475 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.095385 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.095572 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:19.101960 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:19.101979 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:19.101999 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:19.102064 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:19.105916 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.106980 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.106990 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.106992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.107177 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.107934 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.107940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.107950 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.108551 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:19.108950 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:19.109417 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 10:54:19.109723 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":387,"threads_started":1}
I20260928 10:54:19.112517 18603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:19.211409 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 18475
I20260928 10:54:19.230516 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 18614
Sep 28 10:54:19 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:54:19.256321 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:19.231858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.5.227.130:37829)
0928 10:54:19.232524 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:19.232532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:19.232568 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:54:19.232634 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:19.232640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:19.232871 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:19.233176 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:19.235422 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.237583 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.239700 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.240399 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.240581 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:19.246480 (+ 5899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:19.246500 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:19.246518 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:19.246573 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:19.252140 (+ 5567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.253580 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.253594 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.253596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.253807 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.254334 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.254340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.254345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.254753 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:19.255148 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:19.255540 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 10:54:19.255893 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":427,"threads_started":1}
I20260928 10:54:19.258067 18746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:19.363037 18751 heartbeater.cc:503] Master 127.5.227.190:39855 was elected leader, sending a full tablet report...
I20260928 10:54:19.364663 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 18614
W20260928 10:54:19.379611 18381 connection.cc:535] server connection from 127.5.227.130:52583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:54:19.380290 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 18757
Sep 28 10:54:19 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:54:19.406023 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:19.381620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.5.227.131:42551)
0928 10:54:19.382874 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:19.382883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:19.382918 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:54:19.382984 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:19.382992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:19.383263 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:19.383474 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:19.386555 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.387659 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.389430 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:19.390100 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:19.390303 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:19.396786 (+ 6483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:19.396804 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:19.396823 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:19.396871 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:19.401762 (+ 4891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.402919 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.402936 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.402939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.403105 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:19.403553 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:19.403559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:19.403567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:19.404440 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:19.404840 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:19.405225 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 10:54:19.405624 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":786,"threads_started":1}
I20260928 10:54:19.407546 18880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:19.517198 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 18757
I20260928 10:54:19.535903 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 18363
I20260928 10:54:19.537463 18425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:19.658695 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 18363
I20260928 10:54:19.683122 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39855
--webserver_interface=127.5.227.190
--webserver_port=39771
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39855
--ranger_config_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:20.056777 18897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:20.057106 18897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:20.057196 18897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:20.065013 18897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:20.065155 18897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:20.065202 18897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:20.065251 18897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:20.065294 18897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:20.076243 18897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39855
--ranger_config_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=39771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:20.077593 18897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:20.078948 18897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:20.092952 18903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:20.093354 18905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:20.092938 18902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:20.094928 18897 server_base.cc:1034] running on GCE node
I20260928 10:54:20.096138 18897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:20.097837 18897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:20.099068 18897 hybrid_clock.cc:648] HybridClock initialized: now 1790592860099025 us; error 106 us; skew 500 ppm
I20260928 10:54:20.099498 18897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:20 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:20.103168 18897 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:54:20.105437 18897 webserver.cc:460] Webserver started at http://127.5.227.190:39771/ using document root <none> and password file <none>
I20260928 10:54:20.106232 18897 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:20.106401 18897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:54:20 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:20.172277 18897 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:54:20.176436 18913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:20.178452 18897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:54:20.178710 18897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "db93f039c44d41ffa75bfe0e4b20304e"
format_stamp: "Formatted at 2026-09-28 10:54:15 on dist-test-slave-8w8g"
server_key: "797025e8a7ba4f9c6e11df9dcc3a52ad"
server_key_iv: "e24d2705c93bf60adfbe3a5b7d1d8606"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:20.179845 18897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:20.214758 18897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:20.220144 18897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:20.220484 18897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:20.244154 18897 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:39855
I20260928 10:54:20.244131 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:39855 every 8 connection(s)
I20260928 10:54:20.246335 18897 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:54:20.253957 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 18897
I20260928 10:54:20.255398 18965 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:54:20.255601 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:40711
--local_ip_for_outbound_sockets=127.5.227.129
--tserver_master_addrs=127.5.227.190:39855
--webserver_port=43555
--webserver_interface=127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:20.263139 18965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Bootstrap starting.
I20260928 10:54:20.287248 18965 log.cc:824] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:20.297436 18965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:54:20.298030 18965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Bootstrap complete.
I20260928 10:54:20.308207 18965 raft_consensus.cc:348] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:20.309499 18965 raft_consensus.cc:729] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db93f039c44d41ffa75bfe0e4b20304e, State: Initialized, Role: FOLLOWER
I20260928 10:54:20.310474 18965 consensus_queue.cc:260] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:20.310796 18965 raft_consensus.cc:388] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:54:20.310969 18965 raft_consensus.cc:482] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:54:20.311182 18965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:54:20.316221 18965 raft_consensus.cc:504] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:20.316836 18965 leader_election.cc:302] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db93f039c44d41ffa75bfe0e4b20304e; no voters:
I20260928 10:54:20.317520 18965 leader_election.cc:288] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:54:20.317685 18969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:54:20.318943 18969 raft_consensus.cc:686] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [term 2 LEADER]: Becoming Leader. State: Replica: db93f039c44d41ffa75bfe0e4b20304e, State: Running, Role: LEADER
I20260928 10:54:20.319684 18969 consensus_queue.cc:237] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } }
I20260928 10:54:20.320310 18965 sys_catalog.cc:564] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:20.322793 18971 sys_catalog.cc:455] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: SysCatalogTable state changed. Reason: New leader db93f039c44d41ffa75bfe0e4b20304e. Latest consensus state: current_term: 2 leader_uuid: "db93f039c44d41ffa75bfe0e4b20304e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } } }
I20260928 10:54:20.323747 18971 sys_catalog.cc:458] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:20.322856 18970 sys_catalog.cc:455] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db93f039c44d41ffa75bfe0e4b20304e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db93f039c44d41ffa75bfe0e4b20304e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39855 } } }
I20260928 10:54:20.324184 18970 sys_catalog.cc:458] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:20.337998 18976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:54:20.345012 18976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:54:20.346889 18976 catalog_manager.cc:1259] Loaded cluster ID: ba2ba8448d1c460b8f94f5145feadbab
I20260928 10:54:20.347049 18976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:54:20.347472 18965 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 10:54:20.352283 18976 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:54:20.354931 18976 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Loaded TSK: 0
I20260928 10:54:20.365155 18976 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:54:20.734447 18967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:20.734952 18967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:20.735209 18967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:20.749187 18967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:20.749478 18967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:20.749735 18967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:54:20.768491 18967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:40711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=43555
--tserver_master_addrs=127.5.227.190:39855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:20.771420 18967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:20.773579 18967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:20.807883 19003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:20.808718 19002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:20.815069 19005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:20.820572 18967 server_base.cc:1034] running on GCE node
I20260928 10:54:20.821839 18967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:20.829912 18967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:20.831341 18967 hybrid_clock.cc:648] HybridClock initialized: now 1790592860831235 us; error 96 us; skew 500 ppm
I20260928 10:54:20.832046 18967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:20 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:20.837227 18967 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:54:20.856216 18967 webserver.cc:460] Webserver started at http://127.5.227.129:43555/ using document root <none> and password file <none>
I20260928 10:54:20.857820 18967 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:20.858072 18967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:54:20 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:20.910614 18967 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:54:20.915045 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:20.918665 18967 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260928 10:54:20.919346 18967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "0db11ecf9ed745618f61abb9a4e1484f"
format_stamp: "Formatted at 2026-09-28 10:54:17 on dist-test-slave-8w8g"
server_key: "9441f5c424a2f0c52ea0f8b2afcf5bb4"
server_key_iv: "65a5e54f2e8dd85234b23d23a964d79c"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:20.922621 18967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:20.970299 18967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:20.981310 18967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:20.981772 18967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:20.983467 18967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:20.992520 18967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:20.992900 18967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 10:54:20.993251 18967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:20.993458 18967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:21.074688 18967 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:40711
I20260928 10:54:21.074708 19125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:40711 every 8 connection(s)
I20260928 10:54:21.076794 18967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:54:21.085373 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 18967
I20260928 10:54:21.086277 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:37829
--local_ip_for_outbound_sockets=127.5.227.130
--tserver_master_addrs=127.5.227.190:39855
--webserver_port=38155
--webserver_interface=127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:54:21 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:21.113286 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:21.080525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39855 (local address 127.5.227.129:41951)
0928 10:54:21.081428 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:21.081477 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:21.084013 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:21.086062 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:21.086074 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:21.086838 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:21.087722 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:21.087786 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.090481 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.090488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:21.092725 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:21.092754 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.094796 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.094804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:21.095106 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:21.096034 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:21.096071 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:21.099154 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:21.105636 (+ 6482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:21.105651 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:21.105675 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:21.106451 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:21.107698 (+ 1247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:21.107707 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:21.107714 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:21.108245 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:21.109504 (+ 1259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:21.109516 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:21.110312 (+ 796us) client_negotiation.cc:769] Sending connection context
0928 10:54:21.110645 (+ 333us) client_negotiation.cc:240] Negotiation successful
0928 10:54:21.111371 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":396,"threads_started":1}
I20260928 10:54:21.117870 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:21.080678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:41951 (local address 127.5.227.190:39855)
0928 10:54:21.082447 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:21.082466 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:21.082540 (+ 74us) server_negotiation.cc:407] Connection header received
0928 10:54:21.084858 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:21.084881 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:21.085330 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:21.085815 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:21.088083 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.090208 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.092980 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.094133 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.094346 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:21.099628 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:21.099645 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:21.099672 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:21.099772 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:21.105344 (+ 5572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:21.106870 (+ 1526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:21.106883 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:21.106895 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:21.107452 (+ 557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:21.108454 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:21.108462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:21.108468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:21.109306 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:21.113980 (+ 4674us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:21.114434 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 10:54:21.115042 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":1203,"threads_started":1}
I20260928 10:54:21.123495 19126 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39855
I20260928 10:54:21.124063 19126 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:21.125075 19126 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
W20260928 10:54:21.129575 19126 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:39855 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:54:21.129890 19126 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:21.130393 19126 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
I20260928 10:54:21.133932 19126 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:21.134714 19126 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
W20260928 10:54:21.136931 19126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:54:21.585515 19131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:21.585947 19131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:21.586162 19131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:21.598866 19131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:21.599119 19131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:21.599364 19131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:54:21.615057 19131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:37829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=38155
--tserver_master_addrs=127.5.227.190:39855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:21.616612 19131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:21.618139 19131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:21.635967 19146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:21.635989 19144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:21.639534 19143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:21.641952 19131 server_base.cc:1034] running on GCE node
I20260928 10:54:21.642985 19131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:21.644258 19131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:21.645637 19131 hybrid_clock.cc:648] HybridClock initialized: now 1790592861645570 us; error 60 us; skew 500 ppm
I20260928 10:54:21.646031 19131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:21 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:21.650485 19131 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:54:21.652678 19131 webserver.cc:460] Webserver started at http://127.5.227.130:38155/ using document root <none> and password file <none>
I20260928 10:54:21.653914 19131 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:21.654153 19131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:54:21 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:21.698221 19131 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.013s sys 0.003s
I20260928 10:54:21.701735 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:21.704478 19131 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 10:54:21.704771 19131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "5e82fc306fca46a38edd20a38706a39f"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "1b80c13f90f4b7b6436bd3478fd8aea7"
server_key_iv: "b5c403c933d5042a8a4ab44ac176169b"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:21.705929 19131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:21.733686 19131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:21.739985 19131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:21.740758 19131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:21.756302 19131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:21.764998 19131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:21.765256 19131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:54:21.765461 19131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:21.765607 19131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:21.865221 19131 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:37829
I20260928 10:54:21.865279 19270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:37829 every 8 connection(s)
I20260928 10:54:21.867321 19131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:54:21.876900 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19131
I20260928 10:54:21.877966 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.227.148:56045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:42551
--local_ip_for_outbound_sockets=127.5.227.131
--tserver_master_addrs=127.5.227.190:39855
--webserver_port=42405
--webserver_interface=127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:54:21 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:21.923290 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:21.872088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:54151 (local address 127.5.227.190:39855)
0928 10:54:21.893030 (+ 20942us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:21.893212 (+ 182us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:21.896770 (+ 3558us) server_negotiation.cc:407] Connection header received
0928 10:54:21.898607 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:21.898614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:21.898745 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:21.898841 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:21.903195 (+ 4354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.904482 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.908215 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.908922 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.909043 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:21.914668 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:21.914676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:21.914682 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:21.914732 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:21.918608 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:21.919899 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:21.919905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:21.919909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:21.920101 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:21.920756 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:21.920762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:21.920765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:21.921178 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:21.921325 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:21.922437 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0928 10:54:21.922743 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20749,"thread_start_us":24575,"threads_started":1}
I20260928 10:54:21.926499 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:21.872717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39855 (local address 127.5.227.130:54151)
0928 10:54:21.893938 (+ 21221us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:21.894010 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:21.898138 (+ 4128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:21.899568 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:21.899583 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:21.900369 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:21.902188 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:21.902228 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.904770 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.904776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:21.907948 (+ 3172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:21.907976 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:21.909478 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:21.909486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:21.909900 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:21.910995 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:21.911030 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:21.914312 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:21.918864 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:21.918876 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:21.918899 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:21.919671 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:21.920285 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:21.920290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:21.920293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:21.920600 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:21.921423 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:21.921432 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:21.921953 (+ 521us) client_negotiation.cc:769] Sending connection context
0928 10:54:21.923970 (+ 2017us) client_negotiation.cc:240] Negotiation successful
0928 10:54:21.924824 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20811,"thread_start_us":22885,"threads_started":1}
I20260928 10:54:21.929144 19271 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39855
Sep 28 10:54:21 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:21.929553 19271 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:21.930614 19271 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
I20260928 10:54:21.933924 18930 ts_manager.cc:194] Registered new tserver with Master: 5e82fc306fca46a38edd20a38706a39f (127.5.227.130:37829)
I20260928 10:54:21.936771 18930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:54151
I20260928 10:54:22.137135 19126 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:22.137650 19126 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
I20260928 10:54:22.139282 18930 ts_manager.cc:194] Registered new tserver with Master: 0db11ecf9ed745618f61abb9a4e1484f (127.5.227.129:40711)
I20260928 10:54:22.141180 18930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:41951
W20260928 10:54:22.364725 19275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:22.365084 19275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:22.365257 19275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:22.375809 19275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:22.375981 19275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:22.376166 19275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:54:22.389112 19275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.227.148:56045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:42551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=42405
--tserver_master_addrs=127.5.227.190:39855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:22.390600 19275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:22.392047 19275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:22.414328 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:22.419363 19282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:22.425331 19275 server_base.cc:1034] running on GCE node
W20260928 10:54:22.426259 19284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:22.427191 19275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:22.429222 19275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:22.430536 19275 hybrid_clock.cc:648] HybridClock initialized: now 1790592862430446 us; error 59 us; skew 500 ppm
I20260928 10:54:22.431057 19275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:22 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
Sep 28 10:54:22 dist-test-slave-8w8g krb5kdc[17919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592862, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:22.451368 19275 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:54:22.471513 19275 webserver.cc:460] Webserver started at http://127.5.227.131:42405/ using document root <none> and password file <none>
I20260928 10:54:22.473219 19275 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:22.473455 19275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:54:22 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592862, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:22.505189 19275 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:54:22.517905 19292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:22.520368 19275 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.007s sys 0.000s
I20260928 10:54:22.520594 19275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "4fcdbe053a914bce8ca8b6e07a2a9058"
format_stamp: "Formatted at 2026-09-28 10:54:18 on dist-test-slave-8w8g"
server_key: "cf08ce00f3c0f687d833747d6911e11d"
server_key_iv: "c7eb57311c307011bac8d56a6a3fb9d9"
server_key_version: "kuduclusterkey@0"
I20260928 10:54:22.522040 19275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:22.579524 19275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:22.585006 19275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:22.585345 19275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:22.591778 19275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:22.596349 19275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:22.596527 19275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:22.596690 19275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:22.596784 19275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:22.647367 19275 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:42551
I20260928 10:54:22.647475 19404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:42551 every 8 connection(s)
I20260928 10:54:22.649143 19275 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:54:22.652746 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19275
Sep 28 10:54:22 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592862, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:22.697504 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.655293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.5.227.190:39855)
0928 10:54:22.675613 (+ 20320us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:22.675621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:22.676612 (+ 991us) server_negotiation.cc:407] Connection header received
0928 10:54:22.676696 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:22.676701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:22.676791 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:22.676890 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:22.678986 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.680127 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.683784 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.684791 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.684933 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.689187 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:22.689194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:22.689199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:22.689249 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:22.693319 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:22.694789 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:22.694797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:22.694802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:22.694974 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:22.695526 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:22.695531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:22.695534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:22.695932 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:22.696211 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:22.696859 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 10:54:22.697101 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20148,"spinlock_wait_cycles":7552,"thread_start_us":19986,"threads_started":1}
I20260928 10:54:22.707901 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.653524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:49723 (local address 127.5.227.190:39855)
0928 10:54:22.669585 (+ 16061us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:22.669594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:22.681101 (+ 11507us) server_negotiation.cc:407] Connection header received
0928 10:54:22.684326 (+ 3225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:22.684334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:22.684487 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:22.684598 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:22.687826 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.689397 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.692273 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.693032 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.693154 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.699421 (+ 6267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:22.699431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:22.699436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:22.699476 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:22.703304 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:22.704601 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:22.704608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:22.704612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:22.704784 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:22.705448 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:22.705454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:22.705457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:22.705923 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:22.706098 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:22.707144 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0928 10:54:22.707442 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15869,"thread_start_us":15657,"threads_started":1}
I20260928 10:54:22.710412 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.654034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39855 (local address 127.5.227.131:49723)
0928 10:54:22.680925 (+ 26891us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:22.680986 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:22.683637 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:22.685426 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:22.685440 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:22.686265 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:22.687426 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.687463 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.689697 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.689711 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.692028 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.692056 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.693615 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.693623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.693964 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.694897 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:22.694933 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:22.698864 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:22.703598 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:22.703612 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:22.703635 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:22.704402 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:22.704930 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:22.704935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:22.704939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:22.705264 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:22.706093 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:22.706103 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:22.706663 (+ 560us) client_negotiation.cc:769] Sending connection context
0928 10:54:22.708222 (+ 1559us) client_negotiation.cc:240] Negotiation successful
0928 10:54:22.708895 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26532,"thread_start_us":26286,"threads_started":1}
I20260928 10:54:22.713258 19405 heartbeater.cc:347] Connected to a master server at 127.5.227.190:39855
I20260928 10:54:22.713652 19405 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:22.714623 19405 heartbeater.cc:511] Master 127.5.227.190:39855 requested a full tablet report, sending...
I20260928 10:54:22.716528 18930 ts_manager.cc:194] Registered new tserver with Master: 4fcdbe053a914bce8ca8b6e07a2a9058 (127.5.227.131:42551)
I20260928 10:54:22.717658 18930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:49723
I20260928 10:54:22.727144 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:54:22.754968 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.734679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.5.227.190:39855)
0928 10:54:22.734862 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:22.734869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:22.736363 (+ 1494us) server_negotiation.cc:407] Connection header received
0928 10:54:22.736556 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:22.736562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:22.736651 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:22.736782 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:22.738898 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.740094 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.742144 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.742769 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.742911 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.747302 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:22.747310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:22.747314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:22.747369 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:22.751211 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:22.752333 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:22.752339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:22.752343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:22.752537 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:22.753072 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:22.753077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:22.753081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:22.753477 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:22.753736 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:22.754339 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 10:54:22.754619 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:54:22.762753 18930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:54:22.775071 18930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 10:54:22 dist-test-slave-8w8g krb5kdc[17919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for HTTP/127.5.227.148@KRBTEST.COM
I20260928 10:54:22.834800 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.812095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:42551 (local address 127.0.0.1:35944)
0928 10:54:22.821957 (+ 9862us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:22.822125 (+ 168us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:22.822413 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:22.824429 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:22.824440 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:22.824484 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:22.825016 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.825033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.827415 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.827424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.830458 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.830485 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.833433 (+ 2948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.833444 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.833761 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.833840 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 10:54:22.834002 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 10:54:22.834279 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9421,"thread_start_us":9195,"threads_started":1}
I20260928 10:54:22.836252 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.812320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.5.227.131:42551)
0928 10:54:22.823233 (+ 10913us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:22.823246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:22.823288 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:54:22.823415 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:22.823427 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:22.823770 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:22.824001 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:22.825272 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.827192 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.830739 (+ 3547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.832706 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.832890 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.833218 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:22.835543 (+ 2325us) server_negotiation.cc:299] Negotiation successful
0928 10:54:22.835824 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10690,"thread_start_us":11458,"threads_started":1}
I20260928 10:54:22.841763 19340 tablet_service.cc:1467] Processing CreateTablet for tablet bb471192bf974ed296895ebe184ed249 (DEFAULT_TABLE table=test-table [id=7a57c6e26d364d7f894b293d43d7b0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:22.843487 19340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb471192bf974ed296895ebe184ed249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:22.859808 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.822869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:37829 (local address 127.0.0.1:58224)
0928 10:54:22.837086 (+ 14217us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:22.837110 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:22.837261 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:22.844723 (+ 7462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:22.844727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:22.844750 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:22.845066 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.845076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.849554 (+ 4478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.849561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.851986 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.852010 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.858903 (+ 6893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.858911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.859091 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.859147 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 10:54:22.859242 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 10:54:22.859403 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14042,"thread_start_us":13857,"threads_started":1}
I20260928 10:54:22.860673 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.822904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.5.227.130:37829)
0928 10:54:22.841204 (+ 18300us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:22.841213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:22.841245 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:54:22.841352 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:22.841360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:22.841631 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:22.841846 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:22.845509 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.847299 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.852247 (+ 4948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.853880 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.854036 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.854264 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:22.860146 (+ 5882us) server_negotiation.cc:299] Negotiation successful
0928 10:54:22.860362 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18112,"thread_start_us":20991,"threads_started":1}
I20260928 10:54:22.862921 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.816403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.5.227.129:40711)
0928 10:54:22.842557 (+ 26154us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:22.842566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:22.842596 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:54:22.842712 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:22.842720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:22.842945 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:22.843142 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:22.847552 (+ 4410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.849370 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.856950 (+ 7580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.858628 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.861910 (+ 3282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.862296 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:22.862406 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 10:54:22.862605 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25886,"thread_start_us":27670,"threads_started":1}
I20260928 10:54:22.861742 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:22.815715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40711 (local address 127.0.0.1:57082)
0928 10:54:22.835111 (+ 19396us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:22.835138 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:22.835311 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:22.844325 (+ 9014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:22.844331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:22.844361 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:22.844552 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.844562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.854418 (+ 9856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.854425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.856650 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:22.856676 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:22.860958 (+ 4282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:22.860965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:22.861191 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:22.861229 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:54:22.861305 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 10:54:22.861469 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19273,"thread_start_us":27979,"threads_started":1}
I20260928 10:54:22.868176 19206 tablet_service.cc:1467] Processing CreateTablet for tablet bb471192bf974ed296895ebe184ed249 (DEFAULT_TABLE table=test-table [id=7a57c6e26d364d7f894b293d43d7b0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:22.870007 19206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb471192bf974ed296895ebe184ed249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:22.870936 19061 tablet_service.cc:1467] Processing CreateTablet for tablet bb471192bf974ed296895ebe184ed249 (DEFAULT_TABLE table=test-table [id=7a57c6e26d364d7f894b293d43d7b0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:22.872762 19061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb471192bf974ed296895ebe184ed249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:22.899533 19427 tablet_bootstrap.cc:492] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Bootstrap starting.
I20260928 10:54:22.902505 19428 tablet_bootstrap.cc:492] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Bootstrap starting.
I20260928 10:54:22.907435 19427 tablet_bootstrap.cc:654] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:22.912075 19428 tablet_bootstrap.cc:654] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:22.913664 19428 log.cc:824] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:22.908936 19427 log.cc:824] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:22.938087 19427 tablet_bootstrap.cc:492] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: No bootstrap required, opened a new log
I20260928 10:54:22.938431 19427 ts_tablet_manager.cc:1397] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.005s
I20260928 10:54:22.939647 19429 tablet_bootstrap.cc:492] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Bootstrap starting.
I20260928 10:54:22.942401 19271 heartbeater.cc:503] Master 127.5.227.190:39855 was elected leader, sending a full tablet report...
I20260928 10:54:22.945590 19429 tablet_bootstrap.cc:654] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:22.947067 19429 log.cc:824] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:22.953794 19428 tablet_bootstrap.cc:492] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: No bootstrap required, opened a new log
I20260928 10:54:22.954233 19428 ts_tablet_manager.cc:1397] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.011s
I20260928 10:54:22.966288 19427 raft_consensus.cc:348] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:22.967060 19427 raft_consensus.cc:374] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:22.967221 19427 raft_consensus.cc:729] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e82fc306fca46a38edd20a38706a39f, State: Initialized, Role: FOLLOWER
I20260928 10:54:22.976672 19427 consensus_queue.cc:260] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:22.980814 19429 tablet_bootstrap.cc:492] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: No bootstrap required, opened a new log
I20260928 10:54:22.981199 19429 ts_tablet_manager.cc:1397] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.025s
I20260928 10:54:22.986039 19427 ts_tablet_manager.cc:1428] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Time spent starting tablet: real 0.047s user 0.014s sys 0.005s
I20260928 10:54:22.991695 19428 raft_consensus.cc:348] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:22.996766 19428 raft_consensus.cc:374] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:22.996905 19428 raft_consensus.cc:729] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db11ecf9ed745618f61abb9a4e1484f, State: Initialized, Role: FOLLOWER
I20260928 10:54:23.000285 19428 consensus_queue.cc:260] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:23.000401 19429 raft_consensus.cc:348] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:23.000892 19429 raft_consensus.cc:374] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:23.001009 19429 raft_consensus.cc:729] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fcdbe053a914bce8ca8b6e07a2a9058, State: Initialized, Role: FOLLOWER
I20260928 10:54:23.001753 19429 consensus_queue.cc:260] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:23.003521 19405 heartbeater.cc:503] Master 127.5.227.190:39855 was elected leader, sending a full tablet report...
I20260928 10:54:23.004464 19429 ts_tablet_manager.cc:1428] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Time spent starting tablet: real 0.023s user 0.013s sys 0.003s
I20260928 10:54:23.005664 19126 heartbeater.cc:503] Master 127.5.227.190:39855 was elected leader, sending a full tablet report...
I20260928 10:54:23.007083 19428 ts_tablet_manager.cc:1428] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Time spent starting tablet: real 0.053s user 0.014s sys 0.007s
W20260928 10:54:23.080448 19127 tablet.cc:2369] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:54:23.153676 19406 tablet.cc:2369] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:54:23.154472 19272 tablet.cc:2369] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:54:23.241858 19435 raft_consensus.cc:482] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:54:23.242201 19435 raft_consensus.cc:504] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:23.244365 19435 leader_election.cc:288] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db11ecf9ed745618f61abb9a4e1484f (127.5.227.129:40711), 5e82fc306fca46a38edd20a38706a39f (127.5.227.130:37829)
I20260928 10:54:23.263448 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.245258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:35589 (local address 127.5.227.129:40711)
0928 10:54:23.245540 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:23.245547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:23.255593 (+ 10046us) server_negotiation.cc:407] Connection header received
0928 10:54:23.255888 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:23.255894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:23.256004 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:23.256082 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:23.256947 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.258037 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.260837 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.262170 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.262319 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.262399 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:23.263000 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 10:54:23.263161 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 10:54:23.264678 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.244789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40711 (local address 127.5.227.131:35589)
0928 10:54:23.255402 (+ 10613us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:23.255428 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:23.255666 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:23.256311 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:23.256316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:23.256346 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:23.256726 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:23.256740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.258282 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.258287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:23.260597 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:23.260620 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.262607 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.262613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:23.262803 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.262835 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:54:23.264041 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0928 10:54:23.264262 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10425,"mutex_wait_us":39,"thread_start_us":14971,"threads_started":1}
I20260928 10:54:23.266003 19081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb471192bf974ed296895ebe184ed249" candidate_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db11ecf9ed745618f61abb9a4e1484f" is_pre_election: true
I20260928 10:54:23.266613 19081 raft_consensus.cc:2454] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcdbe053a914bce8ca8b6e07a2a9058 in term 0.
I20260928 10:54:23.267590 19296 leader_election.cc:302] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db11ecf9ed745618f61abb9a4e1484f, 4fcdbe053a914bce8ca8b6e07a2a9058; no voters:
I20260928 10:54:23.267903 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.245848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:47289 (local address 127.5.227.130:37829)
0928 10:54:23.246048 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:23.246054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:23.260730 (+ 14676us) server_negotiation.cc:407] Connection header received
0928 10:54:23.260945 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:23.260951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:23.261063 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:23.261127 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:23.261974 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.263041 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.265447 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.266646 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.266794 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.266868 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:23.267471 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 10:54:23.267629 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:54:23.268083 19435 raft_consensus.cc:2781] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:54:23.268271 19435 raft_consensus.cc:482] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:54:23.268394 19435 raft_consensus.cc:3037] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:23.268798 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.245058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:37829 (local address 127.5.227.131:47289)
0928 10:54:23.260566 (+ 15508us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:23.260589 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:23.260767 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:23.261335 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:23.261340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:23.261367 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:23.261751 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:23.261763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.263233 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.263238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:23.265263 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:23.265278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.267078 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.267084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:23.267272 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.267310 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:54:23.268152 (+ 842us) client_negotiation.cc:240] Negotiation successful
0928 10:54:23.268310 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15348,"mutex_wait_us":1,"thread_start_us":15162,"threads_started":1}
I20260928 10:54:23.270995 19226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb471192bf974ed296895ebe184ed249" candidate_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e82fc306fca46a38edd20a38706a39f" is_pre_election: true
I20260928 10:54:23.271623 19226 raft_consensus.cc:2454] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcdbe053a914bce8ca8b6e07a2a9058 in term 0.
I20260928 10:54:23.271986 19435 raft_consensus.cc:504] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:23.273566 19435 leader_election.cc:288] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [CANDIDATE]: Term 1 election: Requested vote from peers 0db11ecf9ed745618f61abb9a4e1484f (127.5.227.129:40711), 5e82fc306fca46a38edd20a38706a39f (127.5.227.130:37829)
I20260928 10:54:23.273613 19081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb471192bf974ed296895ebe184ed249" candidate_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db11ecf9ed745618f61abb9a4e1484f"
I20260928 10:54:23.273860 19081 raft_consensus.cc:3037] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:23.274375 19226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb471192bf974ed296895ebe184ed249" candidate_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e82fc306fca46a38edd20a38706a39f"
I20260928 10:54:23.274585 19226 raft_consensus.cc:3037] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:23.277328 19081 raft_consensus.cc:2454] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcdbe053a914bce8ca8b6e07a2a9058 in term 1.
I20260928 10:54:23.277899 19296 leader_election.cc:302] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db11ecf9ed745618f61abb9a4e1484f, 4fcdbe053a914bce8ca8b6e07a2a9058; no voters:
I20260928 10:54:23.278038 19226 raft_consensus.cc:2454] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcdbe053a914bce8ca8b6e07a2a9058 in term 1.
I20260928 10:54:23.278333 19435 raft_consensus.cc:2781] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:54:23.278975 19435 raft_consensus.cc:686] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 1 LEADER]: Becoming Leader. State: Replica: 4fcdbe053a914bce8ca8b6e07a2a9058, State: Running, Role: LEADER
I20260928 10:54:23.279522 19435 consensus_queue.cc:237] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } }
I20260928 10:54:23.284868 18929 catalog_manager.cc:5495] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fcdbe053a914bce8ca8b6e07a2a9058 (127.5.227.131). New cstate: current_term: 1 leader_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcdbe053a914bce8ca8b6e07a2a9058" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 } health_report { overall_health: UNKNOWN } } }
I20260928 10:54:23.327603 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.321022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.5.227.131:42551)
0928 10:54:23.321388 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:23.321395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:23.321777 (+ 382us) server_negotiation.cc:407] Connection header received
0928 10:54:23.322038 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:23.322042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:23.322120 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:23.322299 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:54:23.322753 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.323506 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.325201 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.325619 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.325765 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.326325 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:54:23.326882 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:54:23.327098 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:23.327168 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 10:54:23.327329 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 10:54:23.337908 19226 raft_consensus.cc:1261] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 1 FOLLOWER]: Refusing update from remote peer 4fcdbe053a914bce8ca8b6e07a2a9058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:23.338020 19081 raft_consensus.cc:1261] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 1 FOLLOWER]: Refusing update from remote peer 4fcdbe053a914bce8ca8b6e07a2a9058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:23.339262 19445 consensus_queue.cc:1035] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db11ecf9ed745618f61abb9a4e1484f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:23.339658 19435 consensus_queue.cc:1035] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e82fc306fca46a38edd20a38706a39f" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:23.356606 19449 mvcc.cc:205] Tried to move back new op lower bound from 7334268368220753920 to 7334268367997636608. Current Snapshot: MvccSnapshot[applied={T|T < 7334268368220753920}]
I20260928 10:54:23.366283 19450 mvcc.cc:205] Tried to move back new op lower bound from 7334268368220753920 to 7334268367997636608. Current Snapshot: MvccSnapshot[applied={T|T < 7334268368220753920}]
I20260928 10:54:23.382716 18929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:54:23.383037 18929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:54:23.387975 18929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db93f039c44d41ffa75bfe0e4b20304e: Sending DeleteTablet for 3 replicas of tablet bb471192bf974ed296895ebe184ed249
I20260928 10:54:23.389816 19061 tablet_service.cc:1514] Processing DeleteTablet for tablet bb471192bf974ed296895ebe184ed249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:23 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:57082
I20260928 10:54:23.389838 19340 tablet_service.cc:1514] Processing DeleteTablet for tablet bb471192bf974ed296895ebe184ed249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:23 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:35944
I20260928 10:54:23.391259 19458 tablet_replica.cc:330] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: stopping tablet replica
I20260928 10:54:23.391822 19458 raft_consensus.cc:2229] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:54:23.391947 19206 tablet_service.cc:1514] Processing DeleteTablet for tablet bb471192bf974ed296895ebe184ed249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:23 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:58224
I20260928 10:54:23.392547 19458 raft_consensus.cc:2258] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:23.395881 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 18967
I20260928 10:54:23.399343 19458 ts_tablet_manager.cc:1905] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:23.402351 19459 tablet_replica.cc:330] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: stopping tablet replica
I20260928 10:54:23.403124 19459 raft_consensus.cc:2229] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:54:23.403842 19459 raft_consensus.cc:2258] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:23.407618 19459 ts_tablet_manager.cc:1905] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:23.408250 19458 ts_tablet_manager.cc:1918] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:23.408459 19458 log.cc:1196] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/bb471192bf974ed296895ebe184ed249
I20260928 10:54:23.409236 19458 ts_tablet_manager.cc:1939] T bb471192bf974ed296895ebe184ed249 P 4fcdbe053a914bce8ca8b6e07a2a9058: Deleting consensus metadata
I20260928 10:54:23.411302 18915 catalog_manager.cc:4841] TS 4fcdbe053a914bce8ca8b6e07a2a9058 (127.5.227.131:42551): tablet bb471192bf974ed296895ebe184ed249 (table test-table [id=7a57c6e26d364d7f894b293d43d7b0b5]) successfully deleted
I20260928 10:54:23.418262 19459 ts_tablet_manager.cc:1918] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:23.418498 19459 log.cc:1196] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/bb471192bf974ed296895ebe184ed249
I20260928 10:54:23.419198 19459 ts_tablet_manager.cc:1939] T bb471192bf974ed296895ebe184ed249 P 0db11ecf9ed745618f61abb9a4e1484f: Deleting consensus metadata
I20260928 10:54:23.419206 19460 tablet_replica.cc:330] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: stopping tablet replica
I20260928 10:54:23.419806 19460 raft_consensus.cc:2229] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:54:23.421440 18917 catalog_manager.cc:4841] TS 0db11ecf9ed745618f61abb9a4e1484f (127.5.227.129:40711): tablet bb471192bf974ed296895ebe184ed249 (table test-table [id=7a57c6e26d364d7f894b293d43d7b0b5]) successfully deleted
I20260928 10:54:23.421783 19460 raft_consensus.cc:2258] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:23.424836 19460 ts_tablet_manager.cc:1905] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:23.435068 19460 ts_tablet_manager.cc:1918] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:23.435405 19460 log.cc:1196] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/bb471192bf974ed296895ebe184ed249
I20260928 10:54:23.436125 19460 ts_tablet_manager.cc:1939] T bb471192bf974ed296895ebe184ed249 P 5e82fc306fca46a38edd20a38706a39f: Deleting consensus metadata
I20260928 10:54:23.436875 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.398645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.5.227.129:40711)
0928 10:54:23.398869 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:23.398876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:23.414986 (+ 16110us) server_negotiation.cc:407] Connection header received
0928 10:54:23.415260 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:23.415266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:23.415360 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:23.415473 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:23.417687 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.418890 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.421141 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.421772 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.421891 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.426947 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:23.426971 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:23.426993 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:23.427060 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:23.432244 (+ 5184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:23.433678 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:23.433691 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:23.433695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:23.433901 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:23.434482 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:23.434489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:23.434492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:23.435031 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:23.435400 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:23.436174 (+ 774us) server_negotiation.cc:299] Negotiation successful
0928 10:54:23.436491 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:54:23.438570 18915 catalog_manager.cc:4841] TS 5e82fc306fca46a38edd20a38706a39f (127.5.227.130:37829): tablet bb471192bf974ed296895ebe184ed249 (table test-table [id=7a57c6e26d364d7f894b293d43d7b0b5]) successfully deleted
I20260928 10:54:23.438673 19121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:23.537508 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 18967
I20260928 10:54:23.556607 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19131
I20260928 10:54:23.578897 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.557606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.5.227.130:37829)
0928 10:54:23.558134 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:23.558138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:23.558156 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:54:23.558208 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:23.558212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:23.558282 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:23.558406 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:23.559901 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.560596 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.561938 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.562352 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.562498 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.565284 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:23.565299 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:23.565316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:23.565352 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:23.568621 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:23.574855 (+ 6234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:23.574867 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:23.574871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:23.575148 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:23.575898 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:23.575906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:23.575909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:23.576492 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:23.576826 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:23.578327 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0928 10:54:23.578600 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260928 10:54:23.579706 19266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:23.695940 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19131
I20260928 10:54:23.717785 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19275
I20260928 10:54:23.737360 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:23.718664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.5.227.131:42551)
0928 10:54:23.718845 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:23.718850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:23.718921 (+ 71us) server_negotiation.cc:407] Connection header received
0928 10:54:23.719279 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:23.719284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:23.719364 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:23.719462 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:23.721508 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.722620 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.724706 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:23.725234 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:23.725370 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:23.729513 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:23.729532 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:23.729550 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:23.729594 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:23.733564 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:23.734702 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:23.734711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:23.734713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:23.734919 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:23.735425 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:23.735429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:23.735431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:23.735914 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:23.736308 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:23.736672 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 10:54:23.736953 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:54:23.738226 19400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:23.863811 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19275
I20260928 10:54:23.885085 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 18897
I20260928 10:54:23.886626 18960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:24.009784 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 18897
I20260928 10:54:24.040326 6030 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 10:54:24.388113 6030 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 10:54:24.388368 6030 mini_ranger.cc:66] Stopping Ranger...
I20260928 10:54:24.736430 6030 mini_ranger.cc:68] Stopped Ranger
2026-09-28 10:54:24.736 UTC [17949] LOG: received smart shutdown request
2026-09-28 10:54:24.743 UTC [17949] LOG: background worker "logical replication launcher" (PID 17958) exited with exit code 1
2026-09-28 10:54:24.743 UTC [17953] LOG: shutting down
2026-09-28 10:54:25.108 UTC [17949] LOG: database system is shut down
2026-09-28T10:54:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (66847 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:54:25 dist-test-slave-8w8g krb5kdc[19474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:54:25 dist-test-slave-8w8g krb5kdc[19474](info): set up 2 sockets
Sep 28 10:54:25 dist-test-slave-8w8g krb5kdc[19474](info): commencing operation
krb5kdc: starting...
W20260928 10:54:26.540939 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:54:26 dist-test-slave-8w8g krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:54:26.820679 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:38681
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:38681,127.5.227.189:41883,127.5.227.188:37289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:27.210391 19487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:27.210722 19487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:27.210887 19487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:27.219655 19487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:27.219913 19487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:27.220008 19487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:27.220275 19487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:27.220371 19487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:27.232179 19487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:38681,127.5.227.189:41883,127.5.227.188:37289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:38681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:27.233896 19487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:27.235541 19487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:27.249900 19493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:27.250373 19492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:27.250406 19495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:27.250864 19487 server_base.cc:1034] running on GCE node
I20260928 10:54:27.251682 19487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:27.253360 19487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:27.254693 19487 hybrid_clock.cc:648] HybridClock initialized: now 1790592867254624 us; error 60 us; skew 500 ppm
I20260928 10:54:27.255298 19487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:27 dist-test-slave-8w8g krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:27.259075 19487 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:54:27.260818 19487 webserver.cc:460] Webserver started at http://127.5.227.190:38209/ using document root <none> and password file <none>
I20260928 10:54:27.261552 19487 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:27.261677 19487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:27.261972 19487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:27.265713 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "73da94b343e941e1bdd679df4254477d"
format_stamp: "Formatted at 2026-09-28 10:54:27 on dist-test-slave-8w8g"
server_key: "3cbd6de729925ef6b85f3cc321064c7f"
server_key_iv: "8086940e675ce20288d390e21dab912e"
server_key_version: "encryptionkey@0"
I20260928 10:54:27.266628 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "73da94b343e941e1bdd679df4254477d"
format_stamp: "Formatted at 2026-09-28 10:54:27 on dist-test-slave-8w8g"
server_key: "3cbd6de729925ef6b85f3cc321064c7f"
server_key_iv: "8086940e675ce20288d390e21dab912e"
server_key_version: "encryptionkey@0"
I20260928 10:54:27.273274 19487 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:54:27.278148 19502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:27.280054 19487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:54:27.280258 19487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "73da94b343e941e1bdd679df4254477d"
format_stamp: "Formatted at 2026-09-28 10:54:27 on dist-test-slave-8w8g"
server_key: "3cbd6de729925ef6b85f3cc321064c7f"
server_key_iv: "8086940e675ce20288d390e21dab912e"
server_key_version: "encryptionkey@0"
I20260928 10:54:27.280465 19487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:27.297070 19487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:27.301930 19487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:27.302213 19487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:27.324460 19487 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:38681
I20260928 10:54:27.324510 19553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:38681 every 8 connection(s)
I20260928 10:54:27.326131 19487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:54:27.328488 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19487
I20260928 10:54:27.328763 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:54:27.329448 6030 external_mini_cluster.cc:1428] Setting key 169747cd03b874dc927516e90b2c6655
I20260928 10:54:27.330763 19554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:27.338037 19554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.227.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.189@KRBTEST.COM" created.
Sep 28 10:54:27 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:27.368371 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.341950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.5.227.190:38681)
0928 10:54:27.344216 (+ 2266us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:27.344227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:27.344289 (+ 62us) server_negotiation.cc:407] Connection header received
0928 10:54:27.346097 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:27.346108 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:27.346486 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:27.346730 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:27.348113 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.350417 (+ 2304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.352480 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.353127 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.353288 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:27.357966 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:27.357981 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:27.358003 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:27.358054 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:27.363042 (+ 4988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:27.364208 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:27.364221 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:27.364227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:27.364454 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:27.365010 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:27.365014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:27.365016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:27.365754 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:27.365942 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:27.366766 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 10:54:27.367084 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":1767,"threads_started":1}
I20260928 10:54:27.368415 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.341961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38681 (local address 127.0.0.1:54812)
0928 10:54:27.343285 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.343359 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:27.345620 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:27.346774 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:27.346781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:27.347451 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:27.347893 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:27.347914 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.350767 (+ 2853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.350772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:27.352299 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:27.352321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.353600 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.353612 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:27.353744 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:27.354740 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:27.354765 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:27.357629 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:27.363295 (+ 5666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:27.363307 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:27.363318 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:27.363997 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:27.364605 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:27.364611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:27.364615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:27.364848 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:27.365928 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:27.365940 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:27.366291 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 10:54:27.366571 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 10:54:27.367329 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":163,"thread_start_us":506,"threads_started":1}
I20260928 10:54:27.371876 19554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:54:27.373169 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.372692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58374)
0928 10:54:27.372877 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.372996 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 10:54:27.373492 19506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
I20260928 10:54:27.375274 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.374805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58380)
0928 10:54:27.374973 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.375070 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 10:54:27.375507 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:41883: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab.
Entry for principal kudu/127.5.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab.
I20260928 10:54:27.392737 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:41883
--webserver_interface=127.5.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab
--principal=kudu/127.5.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:38681,127.5.227.189:41883,127.5.227.188:37289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:27.424880 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } attempt: 1
I20260928 10:54:27.426287 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.425686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58394)
0928 10:54:27.425897 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.426104 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 10:54:27.427561 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.427119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58410)
0928 10:54:27.427285 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.427376 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 10:54:27.427783 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:41883: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
I20260928 10:54:27.496114 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } attempt: 2
I20260928 10:54:27.497676 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.497138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58422)
0928 10:54:27.497371 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.497482 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 10:54:27.498941 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.498442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58434)
0928 10:54:27.498646 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.498755 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 10:54:27.499366 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:41883: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
I20260928 10:54:27.590684 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } attempt: 3
I20260928 10:54:27.592015 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.591478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58448)
0928 10:54:27.591706 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.591879 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 10:54:27.593358 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.592878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58452)
0928 10:54:27.593106 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.593186 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 10:54:27.593652 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:41883: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
I20260928 10:54:27.736934 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } attempt: 4
I20260928 10:54:27.738132 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.737588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58464)
0928 10:54:27.737894 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.737978 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 10:54:27.739274 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.738896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58470)
0928 10:54:27.739053 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.739147 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 10:54:27.739452 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:41883: Network error: Client connection negotiation failed: client connection to 127.5.227.189:41883: connect: Connection refused (error 111)
W20260928 10:54:27.777153 19561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:27.777734 19561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:27.777917 19561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:27.786976 19561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:27.787163 19561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:27.787261 19561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:27.787362 19561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:27.787437 19561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:27.800458 19561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:38681,127.5.227.189:41883,127.5.227.188:37289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab
--principal=kudu/127.5.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:41883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:27.802317 19561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:27.804037 19561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:27.819591 19566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:27.819490 19569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:27.819602 19567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:27.821035 19561 server_base.cc:1034] running on GCE node
I20260928 10:54:27.822048 19561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:27.824373 19561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:27.825695 19561 hybrid_clock.cc:648] HybridClock initialized: now 1790592867825669 us; error 129 us; skew 500 ppm
I20260928 10:54:27.826140 19561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:27 dist-test-slave-8w8g krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:27.838889 19561 init.cc:377] Logged in from keytab as kudu/127.5.227.189@KRBTEST.COM (short username kudu)
I20260928 10:54:27.841002 19561 webserver.cc:460] Webserver started at http://127.5.227.189:41391/ using document root <none> and password file <none>
I20260928 10:54:27.841830 19561 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:27.841995 19561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:27.842335 19561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:27.845778 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data/instance:
uuid: "296fedf1afcb44caa6f4c705bfecd0ec"
format_stamp: "Formatted at 2026-09-28 10:54:27 on dist-test-slave-8w8g"
server_key: "ae87cf6ec9cd8402791c8df515b1fa21"
server_key_iv: "db42cd2b635ecb88a765ef21bba58e85"
server_key_version: "encryptionkey@0"
I20260928 10:54:27.846690 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/wal/instance:
uuid: "296fedf1afcb44caa6f4c705bfecd0ec"
format_stamp: "Formatted at 2026-09-28 10:54:27 on dist-test-slave-8w8g"
server_key: "ae87cf6ec9cd8402791c8df515b1fa21"
server_key_iv: "db42cd2b635ecb88a765ef21bba58e85"
server_key_version: "encryptionkey@0"
I20260928 10:54:27.852752 19561 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:54:27.857805 19576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:27.860044 19561 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.002s
I20260928 10:54:27.860303 19561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/wal
uuid: "296fedf1afcb44caa6f4c705bfecd0ec"
format_stamp: "Formatted at 2026-09-28 10:54:27 on dist-test-slave-8w8g"
server_key: "ae87cf6ec9cd8402791c8df515b1fa21"
server_key_iv: "db42cd2b635ecb88a765ef21bba58e85"
server_key_version: "encryptionkey@0"
I20260928 10:54:27.860538 19561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:27.899538 19561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:27.906044 19561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:27.906399 19561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:27.929597 19561 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.189:41883
I20260928 10:54:27.929574 19627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.189:41883 every 8 connection(s)
I20260928 10:54:27.931428 19561 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
I20260928 10:54:27.936331 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:27.941025 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19561
I20260928 10:54:27.941280 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-1/wal/instance
I20260928 10:54:27.941850 6030 external_mini_cluster.cc:1428] Setting key 84ade544e3e7ae285336a7df3f9bd00b
I20260928 10:54:27.945699 19628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:54:27 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.188@KRBTEST.COM; defaulting to no policy
I20260928 10:54:27.971037 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.949997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.5.227.190:38681)
0928 10:54:27.950584 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:27.950592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:27.952211 (+ 1619us) server_negotiation.cc:407] Connection header received
0928 10:54:27.954618 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:27.954625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:27.954727 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:27.954853 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:27.956749 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.957484 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.959262 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.959696 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.959892 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:27.964089 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:27.964095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:27.964099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:27.964148 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:27.967564 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:27.968530 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:27.968535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:27.968538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:27.968673 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:27.969152 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:27.969156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:27.969158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:27.969445 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:27.969632 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:27.970389 (+ 757us) server_negotiation.cc:299] Negotiation successful
0928 10:54:27.970657 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":276,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.188@KRBTEST.COM" created.
I20260928 10:54:27.972213 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.950828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38681 (local address 127.0.0.1:54820)
0928 10:54:27.951887 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.951933 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:27.954238 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:27.955142 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:27.955151 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:27.955835 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:27.956509 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:27.956531 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.957715 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.957719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:27.959069 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:27.959085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.960015 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.960024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:27.960285 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:27.961252 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:27.961279 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:27.963783 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:27.967827 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:27.967836 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:27.967852 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:27.968355 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:27.968799 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:27.968804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:27.968813 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:27.969016 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:27.969592 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:27.969601 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:27.970004 (+ 403us) client_negotiation.cc:769] Sending connection context
0928 10:54:27.970281 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 10:54:27.970851 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":417,"threads_started":1}
I20260928 10:54:27.974449 19628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:54:27 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:54:27.995177 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.975433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58480)
0928 10:54:27.975697 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:27.975714 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:27.975883 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:27.976982 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:27.976990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:27.977725 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:27.978003 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:27.978021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.979898 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.979909 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:27.981256 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:27.981271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.982445 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.982454 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:27.982765 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:27.983612 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:27.983641 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:27.986814 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:27.991448 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:27.991454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:27.991458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:27.991974 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:27.992795 (+ 821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:27.992800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:27.992803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:27.992992 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:27.994167 (+ 1175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:27.994171 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:27.994299 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 10:54:27.994409 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 10:54:27.994845 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 10:54:27.995603 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.975792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.5.227.189:41883)
0928 10:54:27.976474 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:27.976482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:27.976510 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:54:27.976563 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:27.976569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:27.976786 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:27.976925 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:27.978409 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.979676 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.981454 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:27.982254 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:27.982369 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:27.987160 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:27.987179 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:27.987198 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:27.987234 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:27.991254 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:27.992370 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:27.992385 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:27.992392 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:27.992584 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:27.993157 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:27.993165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:27.993173 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:27.993834 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:27.994390 (+ 556us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:27.994765 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 10:54:27.995074 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":387,"threads_started":1}
I20260928 10:54:27.998528 19628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:54:28.000494 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:27.999827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57142)
0928 10:54:28.000142 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.000314 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260928 10:54:28.000903 19578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.003465 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.002987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57152)
0928 10:54:28.003222 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.003328 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260928 10:54:28.003894 19628 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab.
Entry for principal kudu/127.5.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab.
I20260928 10:54:28.013141 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:37289
--webserver_interface=127.5.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab
--principal=kudu/127.5.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:38681,127.5.227.189:41883,127.5.227.188:37289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:28.038743 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } attempt: 5
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:54:28.053275 19628 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 1
I20260928 10:54:28.053854 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.039800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.5.227.189:41883)
0928 10:54:28.039990 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.039995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.040018 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:54:28.040319 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.040324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.040411 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.040529 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.041535 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.042294 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.043483 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.044040 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.044195 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.047475 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.047480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.047483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.047519 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.050710 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.051532 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.051538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.051547 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.051690 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.052198 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.052203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.052207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.052550 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.052724 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.053197 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.053448 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:54:28.054543 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.039623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58482)
0928 10:54:28.039883 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.039907 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:28.040136 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:28.040555 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:28.040560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:28.041008 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:28.041364 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.041373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.042447 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.042453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.043333 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.043346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.044225 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.044231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.044454 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.045091 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:28.045110 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:28.047248 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:28.050915 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.050922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.050927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.051374 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.051887 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.051893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.051897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.052061 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.052707 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:28.052712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:28.052896 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 10:54:28.053086 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 10:54:28.053427 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 10:54:28.055369 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.053956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57164)
0928 10:54:28.054883 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.055020 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":784}
I20260928 10:54:28.056125 19554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:54:28.056748 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.056237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57176)
0928 10:54:28.056476 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.056554 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260928 10:54:28.057307 19628 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.057446 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.056885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57178)
0928 10:54:28.057142 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.057248 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 10:54:28.058611 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.058261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57186)
0928 10:54:28.058410 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.058501 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 10:54:28.058816 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.110128 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 1
I20260928 10:54:28.111341 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.110870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57192)
0928 10:54:28.111080 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.111177 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 10:54:28.112406 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.112087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57200)
0928 10:54:28.112228 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.112303 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 10:54:28.112707 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.125605 19628 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 2
I20260928 10:54:28.126964 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.126462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57216)
0928 10:54:28.126678 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.126794 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 10:54:28.128294 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.127900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57226)
0928 10:54:28.128053 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.128136 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 10:54:28.128557 19628 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.181018 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 2
I20260928 10:54:28.182473 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.181763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57236)
0928 10:54:28.182187 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.182294 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 10:54:28.183992 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.183484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57248)
0928 10:54:28.183705 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.183856 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 10:54:28.184329 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.219870 19628 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 3
I20260928 10:54:28.221360 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.220635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57260)
0928 10:54:28.220894 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.220994 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 10:54:28.222852 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.222301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57262)
0928 10:54:28.222613 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.222715 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
W20260928 10:54:28.223268 19628 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.290609 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 3
I20260928 10:54:28.292133 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.291563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57266)
0928 10:54:28.291825 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.291929 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 10:54:28.293488 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.293027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57270)
0928 10:54:28.293271 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.293355 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260928 10:54:28.293764 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.366608 19628 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 4
I20260928 10:54:28.367972 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.367283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57276)
0928 10:54:28.367521 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.367719 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 10:54:28.369247 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.368832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57286)
0928 10:54:28.369019 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.369115 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 10:54:28.369751 19628 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
W20260928 10:54:28.405134 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:28.405568 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:28.405738 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:28.414654 19636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:28.414788 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:28.414891 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:28.414984 19636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:28.415077 19636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:28.426987 19636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:38681,127.5.227.189:41883,127.5.227.188:37289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab
--principal=kudu/127.5.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:37289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:28.428694 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:28.430260 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:28.444717 19644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:28.444736 19641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:28.444674 19642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:28.445843 19636 server_base.cc:1034] running on GCE node
I20260928 10:54:28.446871 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:28.448670 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:28.449921 19636 hybrid_clock.cc:648] HybridClock initialized: now 1790592868449834 us; error 87 us; skew 500 ppm
I20260928 10:54:28.450469 19636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592868, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:28.454818 19636 init.cc:377] Logged in from keytab as kudu/127.5.227.188@KRBTEST.COM (short username kudu)
I20260928 10:54:28.456907 19636 webserver.cc:460] Webserver started at http://127.5.227.188:36887/ using document root <none> and password file <none>
I20260928 10:54:28.457780 19636 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:28.457949 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:28.458302 19636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:28.462064 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data/instance:
uuid: "4e2b76c596a5411f9bdd694720d61b07"
format_stamp: "Formatted at 2026-09-28 10:54:28 on dist-test-slave-8w8g"
server_key: "248e86a08ec11642ddbf28daed65acf9"
server_key_iv: "6768d38b04226915aa8ce86465863a39"
server_key_version: "encryptionkey@0"
I20260928 10:54:28.463109 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/wal/instance:
uuid: "4e2b76c596a5411f9bdd694720d61b07"
format_stamp: "Formatted at 2026-09-28 10:54:28 on dist-test-slave-8w8g"
server_key: "248e86a08ec11642ddbf28daed65acf9"
server_key_iv: "6768d38b04226915aa8ce86465863a39"
server_key_version: "encryptionkey@0"
I20260928 10:54:28.470398 19636 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 10:54:28.470995 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 4
I20260928 10:54:28.472313 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.471835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57290)
0928 10:54:28.472025 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.472164 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 10:54:28.474238 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.473368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57292)
0928 10:54:28.473716 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.473833 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260928 10:54:28.474514 19554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:37289: Network error: Client connection negotiation failed: client connection to 127.5.227.188:37289: connect: Connection refused (error 111)
I20260928 10:54:28.475668 19651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:28.477478 19636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:54:28.477733 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/wal
uuid: "4e2b76c596a5411f9bdd694720d61b07"
format_stamp: "Formatted at 2026-09-28 10:54:28 on dist-test-slave-8w8g"
server_key: "248e86a08ec11642ddbf28daed65acf9"
server_key_iv: "6768d38b04226915aa8ce86465863a39"
server_key_version: "encryptionkey@0"
I20260928 10:54:28.477967 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:28.499408 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:28.505456 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:28.505735 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:28.526348 19636 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.188:37289
I20260928 10:54:28.526371 19702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.188:37289 every 8 connection(s)
I20260928 10:54:28.528138 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
I20260928 10:54:28.530987 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19636
I20260928 10:54:28.531201 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592619723274-6030-0/minicluster-data/master-2/wal/instance
I20260928 10:54:28.531713 6030 external_mini_cluster.cc:1428] Setting key 0ea4ac8aa4eb3c68f79502f0c74f86d3
I20260928 10:54:28.533162 19703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:28.544822 19703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592868, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:28.563935 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.534184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.5.227.188:37289)
0928 10:54:28.535087 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.535103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.535160 (+ 57us) server_negotiation.cc:407] Connection header received
0928 10:54:28.537100 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.537135 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.537794 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.538327 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.540263 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.542765 (+ 2502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.544844 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.546294 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.546505 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.553874 (+ 7369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.553884 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.553898 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.553947 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.558529 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.559488 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.559500 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.559512 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.559833 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.560510 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.560514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.560517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.561197 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.561663 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.562205 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.562710 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":374,"threads_started":1}
I20260928 10:54:28.567575 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.548786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38681 (local address 127.0.0.1:54824)
0928 10:54:28.550078 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.550161 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:28.550409 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:28.551664 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:28.551675 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:28.552669 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:28.553000 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.553012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.554672 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.554683 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.555999 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.556015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.556821 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.556828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.557011 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.557674 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:28.557697 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:28.560713 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:28.564224 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.564237 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.564244 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.564960 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.565568 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.565573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.565577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.565771 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.566357 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:28.566365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:28.566563 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 10:54:28.566745 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 10:54:28.567130 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":634,"threads_started":1}
I20260928 10:54:28.567575 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.548874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.5.227.190:38681)
0928 10:54:28.550937 (+ 2063us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.550946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.550971 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:54:28.551163 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.551174 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.551307 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.551424 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.553379 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.554474 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.556175 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.556601 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.556752 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.560946 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.560952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.560955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.561001 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.564031 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.565161 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.565167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.565170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.565397 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.565915 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.565919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.565922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.566220 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.566406 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.566881 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.567131 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1907,"thread_start_us":1782,"threads_started":1}
I20260928 10:54:28.569483 19703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592868, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:54:28.589483 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.568526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.5.227.190:38681)
0928 10:54:28.568760 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.568769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.568789 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:54:28.568861 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.568865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.568980 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.569215 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.571333 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.572227 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.575376 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.576287 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.576441 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.582557 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.582567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.582576 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.582630 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.586510 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.587335 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.587340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.587342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.587491 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.587985 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.587989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.587991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.588289 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.588587 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.588910 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.589168 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:54:28.592223 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.570232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:41883 (local address 127.0.0.1:58490)
0928 10:54:28.570411 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.570431 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:28.570698 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:28.577333 (+ 6635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:28.577343 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:28.577790 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:28.577951 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.577962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.579878 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.579883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.580868 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.580885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.581962 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.581967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.582372 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.583104 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:28.583133 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:28.585831 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:28.589573 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.589579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.589583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.590018 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.590447 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.590452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.590456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.590613 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.591146 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:28.591151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:28.591313 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 10:54:28.591468 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 10:54:28.591830 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 10:54:28.592319 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.570388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.5.227.189:41883)
0928 10:54:28.576969 (+ 6581us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.576974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.576992 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:54:28.577057 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.577061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.577127 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.577228 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.578245 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.579647 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.581065 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.581770 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.581917 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.586101 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.586107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.586111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.586153 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.589157 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.590157 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.590163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.590167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.590316 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.590728 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.590738 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.590742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.591016 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.591169 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.591580 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.591839 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6460,"thread_start_us":6358,"threads_started":1}
I20260928 10:54:28.593195 19703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592868, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:54:28.611919 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.594145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.5.227.188:37289)
0928 10:54:28.594316 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.594321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.594610 (+ 289us) server_negotiation.cc:407] Connection header received
0928 10:54:28.594794 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.594799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.594863 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.594957 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.595690 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.596416 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.597585 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.598369 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.598532 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.603154 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.603159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.603163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.603199 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.607603 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.608542 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.608548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.608552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.608713 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.609775 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.609783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.609787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.610211 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.610405 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.610996 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.611406 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:54:28.612027 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.591942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.5.227.189:41883)
0928 10:54:28.592646 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.592652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.592674 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:54:28.592724 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.592731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.592816 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.592956 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.594678 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.595578 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.596961 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.597416 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.597559 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.604354 (+ 6795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.604364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.604370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.604432 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.608700 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.609507 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.609512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.609515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.609691 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.610134 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.610139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.610142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.610511 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.610752 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.611320 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.611597 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":693,"threads_started":1}
I20260928 10:54:28.612403 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.594374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57304)
0928 10:54:28.594515 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.594534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:28.594678 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:28.594974 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:28.594979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:28.595416 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:28.595549 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.595557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.596566 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.596572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.597448 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.597459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.598607 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.598614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.598845 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.599527 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:28.599554 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:28.602913 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:28.607858 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.607863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.607866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.608382 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.609505 (+ 1123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.609508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.609511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.609653 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.610424 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:28.610428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:28.610691 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 10:54:28.610890 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 10:54:28.611704 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":184}
I20260928 10:54:28.618286 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:54:28.619956 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: Bootstrap starting.
I20260928 10:54:28.625746 19703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:28.628969 19703 log.cc:824] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:28.644629 19703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: No bootstrap required, opened a new log
I20260928 10:54:28.653388 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.626430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.5.227.190:38681)
0928 10:54:28.626670 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.626677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.628575 (+ 1898us) server_negotiation.cc:407] Connection header received
0928 10:54:28.628987 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.628995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.629124 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.629221 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.632006 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.633187 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.635923 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.636990 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.637244 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.642900 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.642908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.642913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.642962 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.648279 (+ 5317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.649750 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.649756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.649759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.649936 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.650644 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.650650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.650654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.651250 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.651512 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.652469 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.652851 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:54:28.654215 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.626614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.5.227.188:37289)
0928 10:54:28.626850 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.626859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.627689 (+ 830us) server_negotiation.cc:407] Connection header received
0928 10:54:28.627950 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.627959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.628070 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.628177 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.631378 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.632846 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.635912 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.636991 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.637245 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.642906 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.642913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.642919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.642962 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.648469 (+ 5507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.649739 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.649746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.649749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.649957 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.650641 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.650647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.650651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.651214 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.651515 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.653236 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.653544 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:54:28.657019 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.629008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.5.227.189:41883)
0928 10:54:28.629563 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.629569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.633952 (+ 4383us) server_negotiation.cc:407] Connection header received
0928 10:54:28.634693 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.634702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.634836 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.634937 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.641530 (+ 6593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.642636 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.644680 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.645315 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.645440 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.649016 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.649023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.649028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.649077 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.652779 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.653894 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.653902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.653906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.654198 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.654868 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.654874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.654877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.655376 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.655845 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.656377 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.656683 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377}
I20260928 10:54:28.658110 19703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.658860 19703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:28.659053 19703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2b76c596a5411f9bdd694720d61b07, State: Initialized, Role: FOLLOWER
I20260928 10:54:28.659881 19703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.660985 19719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:28.661402 19719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:54:28.662279 19703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:28.669024 19628 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 5
W20260928 10:54:28.674218 19730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:54:28.674521 19730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
I20260928 10:54:28.670795 19636 master_runner.cc:186] Error getting master registration for 127.5.227.190:38681: OK, instance_id { permanent_uuid: "73da94b343e941e1bdd679df4254477d" instance_seqno: 1790592867317890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:54:28.685280 19636 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:54:28.692588 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.669748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57318)
0928 10:54:28.669987 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.670008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:28.670145 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:28.670695 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:28.670699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:28.671200 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:28.671445 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.671458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.673685 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.673693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.675400 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.675421 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.676737 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.676745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.677021 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.678244 (+ 1223us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:28.678271 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:28.681477 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:28.688723 (+ 7246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.688730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.688735 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.689367 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.689885 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.689891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.689896 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.690082 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.691037 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:28.691042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:28.691385 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 10:54:28.691555 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 10:54:28.691934 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 10:54:28.693482 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.670238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.5.227.188:37289)
0928 10:54:28.670408 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.670414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.670431 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:54:28.670486 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.670489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.670549 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.670665 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.671659 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.673409 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.675613 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.676336 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.676487 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.681857 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.681865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.681871 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.681928 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.688515 (+ 6587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.689530 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.689535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.689537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.689738 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.690260 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.690266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.690269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.690771 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.690943 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.692628 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.692924 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:54:28.699240 19628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: Bootstrap starting.
I20260928 10:54:28.704192 19628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:28.705698 19628 log.cc:824] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:28.710110 19628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: No bootstrap required, opened a new log
I20260928 10:54:28.718982 19628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.719369 19628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:28.719519 19628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296fedf1afcb44caa6f4c705bfecd0ec, State: Initialized, Role: FOLLOWER
I20260928 10:54:28.720330 19628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.721493 19733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:28.722142 19733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:54:28.723206 19628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [sys.catalog]: configured and running, proceeding with master startup.
W20260928 10:54:28.737957 19744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:54:28.738234 19744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:54:28.732033 19561 master_runner.cc:186] Error getting master registration for 127.5.227.190:38681: OK, instance_id { permanent_uuid: "73da94b343e941e1bdd679df4254477d" instance_seqno: 1790592867317890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:54:28.748890 19561 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:54:28.751801 19554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } attempt: 5
W20260928 10:54:28.758117 19657 tablet.cc:2369] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 10:54:28 dist-test-slave-8w8g krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
I20260928 10:54:28.774133 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.752596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:37289 (local address 127.0.0.1:57322)
0928 10:54:28.752951 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:28.752973 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:28.753138 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:28.753552 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:28.753558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:28.754241 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:28.754488 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.754507 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.756208 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.756216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.758017 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:28.758040 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.759383 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.759391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:28.759617 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.760452 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:28.760480 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:28.763665 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:28.770362 (+ 6697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.770369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.770373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.770998 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.771617 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:28.771623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:28.771627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:28.771865 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:28.772909 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:28.772915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:28.773097 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 10:54:28.773273 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 10:54:28.773626 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260928 10:54:28.774387 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:28.752729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.5.227.188:37289)
0928 10:54:28.752923 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:28.752929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:28.753060 (+ 131us) server_negotiation.cc:407] Connection header received
0928 10:54:28.753282 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:28.753288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:28.753366 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:28.753488 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:28.754737 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.756008 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.758280 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:28.759145 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:28.759313 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:28.764165 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:28.764183 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:28.764190 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:28.764251 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:28.770038 (+ 5787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.771224 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.771231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.771235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.771435 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:28.772006 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:28.772013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:28.772017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:28.772745 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:28.772970 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:28.773453 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 10:54:28.773753 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:54:28.781323 19554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d: Bootstrap starting.
I20260928 10:54:28.789189 19554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:28.790967 19554 log.cc:824] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:28.795428 19554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d: No bootstrap required, opened a new log
I20260928 10:54:28.808740 19554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.809274 19554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:28.809412 19554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73da94b343e941e1bdd679df4254477d, State: Initialized, Role: FOLLOWER
I20260928 10:54:28.810168 19554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.811375 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:28.811920 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:54:28.812952 19554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:28.818684 19747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:54:28.819643 19747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.827404 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73da94b343e941e1bdd679df4254477d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" is_pre_election: true
I20260928 10:54:28.828153 19603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73da94b343e941e1bdd679df4254477d in term 0.
I20260928 10:54:28.829074 19506 leader_election.cc:302] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296fedf1afcb44caa6f4c705bfecd0ec, 73da94b343e941e1bdd679df4254477d; no voters:
I20260928 10:54:28.829986 19748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:54:28.830241 19748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:54:28.830416 19748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:28.831080 19678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73da94b343e941e1bdd679df4254477d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2b76c596a5411f9bdd694720d61b07" is_pre_election: true
I20260928 10:54:28.831813 19678 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73da94b343e941e1bdd679df4254477d in term 0.
I20260928 10:54:28.832590 19747 leader_election.cc:288] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296fedf1afcb44caa6f4c705bfecd0ec (127.5.227.189:41883), 4e2b76c596a5411f9bdd694720d61b07 (127.5.227.188:37289)
I20260928 10:54:28.836459 19748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.838696 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73da94b343e941e1bdd679df4254477d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296fedf1afcb44caa6f4c705bfecd0ec"
I20260928 10:54:28.839079 19603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:28.847357 19603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73da94b343e941e1bdd679df4254477d in term 1.
I20260928 10:54:28.851701 19506 leader_election.cc:302] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296fedf1afcb44caa6f4c705bfecd0ec, 73da94b343e941e1bdd679df4254477d; no voters:
I20260928 10:54:28.852258 19747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:54:28.858521 19678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73da94b343e941e1bdd679df4254477d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2b76c596a5411f9bdd694720d61b07"
I20260928 10:54:28.858938 19678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:28.865188 19678 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73da94b343e941e1bdd679df4254477d in term 1.
I20260928 10:54:28.866832 19748 leader_election.cc:288] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [CANDIDATE]: Term 1 election: Requested vote from peers 296fedf1afcb44caa6f4c705bfecd0ec (127.5.227.189:41883), 4e2b76c596a5411f9bdd694720d61b07 (127.5.227.188:37289)
I20260928 10:54:28.879323 19747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [term 1 LEADER]: Becoming Leader. State: Replica: 73da94b343e941e1bdd679df4254477d, State: Running, Role: LEADER
I20260928 10:54:28.880434 19747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } }
I20260928 10:54:28.889313 19748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73da94b343e941e1bdd679df4254477d. Latest consensus state: current_term: 1 leader_uuid: "73da94b343e941e1bdd679df4254477d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:28.889881 19748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:28.895284 19756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:54:28.908227 19756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 10:54:28.909394 19582 tablet.cc:2369] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:54:28.950417 19678 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [term 1 FOLLOWER]: Refusing update from remote peer 73da94b343e941e1bdd679df4254477d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:28.951951 19748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:28.960852 19603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [term 1 FOLLOWER]: Refusing update from remote peer 73da94b343e941e1bdd679df4254477d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:28.962299 19748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [LEADER]: Connected to new peer: Peer: permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:28.977236 19719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73da94b343e941e1bdd679df4254477d. Latest consensus state: current_term: 1 leader_uuid: "73da94b343e941e1bdd679df4254477d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:28.977663 19719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:54:28.985569 19762 mvcc.cc:205] Tried to move back new op lower bound from 7334268391174037504 to 7334268390943281152. Current Snapshot: MvccSnapshot[applied={T|T < 7334268391174037504}]
I20260928 10:54:28.988281 19733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73da94b343e941e1bdd679df4254477d. Latest consensus state: current_term: 1 leader_uuid: "73da94b343e941e1bdd679df4254477d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:28.988765 19733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:54:28.994875 19764 mvcc.cc:205] Tried to move back new op lower bound from 7334268391174037504 to 7334268390943281152. Current Snapshot: MvccSnapshot[applied={T|T < 7334268391174037504}]
I20260928 10:54:28.996378 19719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73da94b343e941e1bdd679df4254477d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:28.996677 19719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:54:29.000597 19733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73da94b343e941e1bdd679df4254477d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:29.000934 19733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:54:29.004974 19756 catalog_manager.cc:1347] Generated new cluster ID: e5dab1008e0749ab9b55c5b313ad67cf
I20260928 10:54:29.004757 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73da94b343e941e1bdd679df4254477d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:29.005187 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:29.005106 19756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:54:29.005522 19748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73da94b343e941e1bdd679df4254477d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73da94b343e941e1bdd679df4254477d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38681 } } peers { permanent_uuid: "296fedf1afcb44caa6f4c705bfecd0ec" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 41883 } } peers { permanent_uuid: "4e2b76c596a5411f9bdd694720d61b07" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 37289 } } }
I20260928 10:54:29.005769 19748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:29.036201 19765 mvcc.cc:205] Tried to move back new op lower bound from 7334268391174037504 to 7334268390943281152. Current Snapshot: MvccSnapshot[applied={T|T < 7334268391174037504}]
I20260928 10:54:29.081405 19756 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:54:29.084484 19756 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:54:29.121426 19756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73da94b343e941e1bdd679df4254477d: Generated new TSK 0
I20260928 10:54:29.122859 19756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:54:29.676836 19730 catalog_manager.cc:1259] Loaded cluster ID: e5dab1008e0749ab9b55c5b313ad67cf
I20260928 10:54:29.676951 19730 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: loading cluster ID for follower catalog manager: success
I20260928 10:54:29.680167 19730 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: acquiring CA information for follower catalog manager: success
I20260928 10:54:29.682492 19730 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e2b76c596a5411f9bdd694720d61b07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:54:29.740687 19744 catalog_manager.cc:1259] Loaded cluster ID: e5dab1008e0749ab9b55c5b313ad67cf
I20260928 10:54:29.740805 19744 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: loading cluster ID for follower catalog manager: success
I20260928 10:54:29.743584 19744 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: acquiring CA information for follower catalog manager: success
I20260928 10:54:29.745770 19744 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 296fedf1afcb44caa6f4c705bfecd0ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:54:29.766192 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19487
I20260928 10:54:29.767627 19549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:29.886660 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19487
I20260928 10:54:29.909799 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19561
I20260928 10:54:29.911032 19623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:30.028009 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19561
I20260928 10:54:30.049590 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19636
I20260928 10:54:30.050936 19698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:30.169818 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19636
2026-09-28T10:54:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4907 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:54:30 dist-test-slave-8w8g krb5kdc[19781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:54:30 dist-test-slave-8w8g krb5kdc[19781](info): set up 2 sockets
Sep 28 10:54:30 dist-test-slave-8w8g krb5kdc[19781](info): commencing operation
krb5kdc: starting...
W20260928 10:54:31.417368 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:54:31 dist-test-slave-8w8g krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:54:31.773413 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:42349
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:42349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:32.166806 19794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:32.167179 19794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:32.167340 19794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:32.175357 19794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:32.175508 19794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:32.175557 19794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:32.175695 19794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:32.175791 19794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:32.186545 19794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:42349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:42349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:32.188192 19794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:32.189682 19794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:32.204983 19799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:32.204923 19800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:32.205426 19802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:32.205902 19794 server_base.cc:1034] running on GCE node
I20260928 10:54:32.206943 19794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:32.208720 19794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:32.210094 19794 hybrid_clock.cc:648] HybridClock initialized: now 1790592872209992 us; error 95 us; skew 500 ppm
I20260928 10:54:32.210526 19794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:32 dist-test-slave-8w8g krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:32.214321 19794 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:54:32.216255 19794 webserver.cc:460] Webserver started at http://127.5.227.190:36749/ using document root <none> and password file <none>
I20260928 10:54:32.216992 19794 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:32.217108 19794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:32.217386 19794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:32.221876 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "a6e75cfdfdc0494088f5a427e887dfbe"
format_stamp: "Formatted at 2026-09-28 10:54:32 on dist-test-slave-8w8g"
server_key: "cacd8d48ccc1cada02bcf6d036b6c7f5"
server_key_iv: "d6bd3eae4bb696c7da42069ceba864d8"
server_key_version: "encryptionkey@0"
I20260928 10:54:32.222913 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "a6e75cfdfdc0494088f5a427e887dfbe"
format_stamp: "Formatted at 2026-09-28 10:54:32 on dist-test-slave-8w8g"
server_key: "cacd8d48ccc1cada02bcf6d036b6c7f5"
server_key_iv: "d6bd3eae4bb696c7da42069ceba864d8"
server_key_version: "encryptionkey@0"
I20260928 10:54:32.229480 19794 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260928 10:54:32.233822 19809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:32.235448 19794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:54:32.235632 19794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "a6e75cfdfdc0494088f5a427e887dfbe"
format_stamp: "Formatted at 2026-09-28 10:54:32 on dist-test-slave-8w8g"
server_key: "cacd8d48ccc1cada02bcf6d036b6c7f5"
server_key_iv: "d6bd3eae4bb696c7da42069ceba864d8"
server_key_version: "encryptionkey@0"
I20260928 10:54:32.235945 19794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:32.266808 19794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:32.272368 19794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:32.272665 19794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:32.296684 19794 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:42349
I20260928 10:54:32.296681 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:42349 every 8 connection(s)
I20260928 10:54:32.298559 19794 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:54:32.303016 19861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:32.305305 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19794
I20260928 10:54:32.305588 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:54:32.306321 6030 external_mini_cluster.cc:1428] Setting key e0e7a762e6ebe0f02896dcfa1c9ceddf
I20260928 10:54:32.316546 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe: Bootstrap starting.
I20260928 10:54:32.320180 19861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:32.321353 19861 log.cc:824] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:54:32 dist-test-slave-8w8g krb5kdc[19781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:32.325174 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe: No bootstrap required, opened a new log
I20260928 10:54:32.333766 19861 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42349 } }
I20260928 10:54:32.334292 19861 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:32.334426 19861 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e75cfdfdc0494088f5a427e887dfbe, State: Initialized, Role: FOLLOWER
I20260928 10:54:32.335275 19861 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42349 } }
I20260928 10:54:32.335510 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:32.308555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.5.227.190:42349)
0928 10:54:32.309618 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:32.309658 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:32.309714 (+ 56us) server_negotiation.cc:407] Connection header received
0928 10:54:32.311699 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:32.311798 (+ 99us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:32.312309 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:32.312790 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:32.314684 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:32.316464 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:32.318543 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:32.319623 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:32.319903 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:32.325672 (+ 5769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:32.325684 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:32.325705 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:32.325789 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:32.329775 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:32.330808 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:32.330817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:32.330825 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:32.331052 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:32.331602 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:32.331606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:32.331608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:32.332412 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:32.332843 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:32.333368 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 10:54:32.333922 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":496,"threads_started":1}
I20260928 10:54:32.335587 19861 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:54:32.335796 19861 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:54:32.336045 19861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:32.339020 19861 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42349 } }
I20260928 10:54:32.339643 19861 leader_election.cc:302] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e75cfdfdc0494088f5a427e887dfbe; no voters:
I20260928 10:54:32.340543 19861 leader_election.cc:288] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:54:32.340701 19866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:54:32.341292 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:54:32.343139 19861 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:32.344099 19866 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [term 1 LEADER]: Becoming Leader. State: Replica: a6e75cfdfdc0494088f5a427e887dfbe, State: Running, Role: LEADER
I20260928 10:54:32.344801 19866 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42349 } }
I20260928 10:54:32.349370 19867 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42349 } } }
I20260928 10:54:32.349361 19868 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e75cfdfdc0494088f5a427e887dfbe. Latest consensus state: current_term: 1 leader_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e75cfdfdc0494088f5a427e887dfbe" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 42349 } } }
I20260928 10:54:32.349888 19867 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:32.349902 19868 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:32.351804 19876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:54:32.358564 19876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:54:32.369562 19876 catalog_manager.cc:1347] Generated new cluster ID: dfdca908c7794170a5a4703963acf7c9
I20260928 10:54:32.369769 19876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:54:32.380939 19876 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:54:32.383226 19876 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:54:32.397344 19876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6e75cfdfdc0494088f5a427e887dfbe: Generated new TSK 0
I20260928 10:54:32.398535 19876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:54:32.457146 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19794
I20260928 10:54:32.458601 19856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:32.569160 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19794
2026-09-28T10:54:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2394 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:54:32 dist-test-slave-8w8g krb5kdc[19889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:54:32 dist-test-slave-8w8g krb5kdc[19889](info): set up 2 sockets
Sep 28 10:54:32 dist-test-slave-8w8g krb5kdc[19889](info): commencing operation
krb5kdc: starting...
W20260928 10:54:33.760818 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:54:33 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:54:34.032327 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44379
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:34.413021 19902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:34.413411 19902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:34.413578 19902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:34.423710 19902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:34.423966 19902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:34.424474 19902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:34.424721 19902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:34.424837 19902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:34.436734 19902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:34.438253 19902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:34.439757 19902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:34.453783 19908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:34.453743 19907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:34.453722 19910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:34.456290 19902 server_base.cc:1034] running on GCE node
I20260928 10:54:34.457621 19902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:34.459793 19902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:34.461171 19902 hybrid_clock.cc:648] HybridClock initialized: now 1790592874461127 us; error 132 us; skew 500 ppm
I20260928 10:54:34.461704 19902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:34 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:34.470265 19902 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:54:34.472368 19902 webserver.cc:460] Webserver started at http://127.5.227.190:36719/ using document root <none> and password file <none>
I20260928 10:54:34.473160 19902 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:34.473317 19902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:34.473654 19902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:34.477526 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "39346fd2b3bc46b2a5fc38bfebc979de"
format_stamp: "Formatted at 2026-09-28 10:54:34 on dist-test-slave-8w8g"
server_key: "f54af56296eb2a8a71d240a12704a731"
server_key_iv: "45f9877e5bf35b495d3aeb6b9bbfe854"
server_key_version: "encryptionkey@0"
I20260928 10:54:34.479367 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "39346fd2b3bc46b2a5fc38bfebc979de"
format_stamp: "Formatted at 2026-09-28 10:54:34 on dist-test-slave-8w8g"
server_key: "f54af56296eb2a8a71d240a12704a731"
server_key_iv: "45f9877e5bf35b495d3aeb6b9bbfe854"
server_key_version: "encryptionkey@0"
I20260928 10:54:34.487385 19902 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:54:34.491956 19917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:34.493772 19902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:54:34.494017 19902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "39346fd2b3bc46b2a5fc38bfebc979de"
format_stamp: "Formatted at 2026-09-28 10:54:34 on dist-test-slave-8w8g"
server_key: "f54af56296eb2a8a71d240a12704a731"
server_key_iv: "45f9877e5bf35b495d3aeb6b9bbfe854"
server_key_version: "encryptionkey@0"
I20260928 10:54:34.494279 19902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:34.521546 19902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:34.528348 19902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:34.528669 19902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:34.551380 19902 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:44379
I20260928 10:54:34.551417 19968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:44379 every 8 connection(s)
I20260928 10:54:34.552986 19902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:54:34.557503 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:34.561564 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19902
I20260928 10:54:34.561847 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:54:34.562644 6030 external_mini_cluster.cc:1428] Setting key df60df48bcc100a05bf86a8b0d2e8d1b
I20260928 10:54:34.571187 19969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Bootstrap starting.
I20260928 10:54:34.575345 19969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:34.577044 19969 log.cc:824] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:54:34 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:34.582374 19969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: No bootstrap required, opened a new log
I20260928 10:54:34.594964 19969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:34.595604 19969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:34.595716 19969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39346fd2b3bc46b2a5fc38bfebc979de, State: Initialized, Role: FOLLOWER
I20260928 10:54:34.596629 19969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:34.596933 19969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:54:34.597105 19969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:54:34.597254 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:34.565443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.5.227.190:44379)
0928 10:54:34.566342 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:34.566355 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:34.566398 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:54:34.568512 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:34.568537 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:34.568976 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:34.569490 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:34.571460 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:34.573577 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:34.576543 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:34.577705 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:34.577943 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:34.583931 (+ 5988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:34.583942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:34.583964 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:34.584024 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:34.589121 (+ 5097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:34.590345 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:34.590353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:34.590365 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:34.590622 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:34.591142 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:34.591146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:34.591149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:34.592028 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:34.594590 (+ 2562us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:34.594905 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 10:54:34.595462 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":430,"threads_started":1}
I20260928 10:54:34.597326 19969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:34.601507 19969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:34.602329 19969 leader_election.cc:302] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39346fd2b3bc46b2a5fc38bfebc979de; no voters:
I20260928 10:54:34.603228 19969 leader_election.cc:288] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:54:34.603474 19974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:54:34.606263 19969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:34.607751 19974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 LEADER]: Becoming Leader. State: Replica: 39346fd2b3bc46b2a5fc38bfebc979de, State: Running, Role: LEADER
I20260928 10:54:34.608536 19974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:34.612085 19975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } } }
I20260928 10:54:34.612607 19975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:34.615485 19976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39346fd2b3bc46b2a5fc38bfebc979de. Latest consensus state: current_term: 1 leader_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } } }
I20260928 10:54:34.616206 19976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:34.622089 19985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:54:34.628228 19985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:54:34.638432 19985 catalog_manager.cc:1347] Generated new cluster ID: 2820cc864c0f424aa06e8f2dfd9ffac7
I20260928 10:54:34.638608 19985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:54:34.653829 19985 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:54:34.655314 19985 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:54:34.662273 19985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Generated new TSK 0
I20260928 10:54:34.663596 19985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:54:34.745716 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:44379
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:35.119163 19995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:35.119553 19995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:35.119767 19995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:35.127918 19995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:35.128098 19995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:35.128290 19995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:54:35.138896 19995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:35.140911 19995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:35.142625 19995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:35.160200 20003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:35.160323 20000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:35.160446 20001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:35.161562 19995 server_base.cc:1034] running on GCE node
I20260928 10:54:35.162570 19995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:35.164330 19995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:35.165628 19995 hybrid_clock.cc:648] HybridClock initialized: now 1790592875165553 us; error 63 us; skew 500 ppm
I20260928 10:54:35.166038 19995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:35 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:35.170732 19995 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:54:35.172739 19995 webserver.cc:460] Webserver started at http://127.5.227.129:33421/ using document root <none> and password file <none>
I20260928 10:54:35.173552 19995 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:35.173719 19995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:35.174043 19995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:35.178100 19995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "bef7295215fb4179818f49d684029947"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "fb7344a41595620147dde7e0ee4e4cc7"
server_key_iv: "8d2196114f3b8f49fa3f3a782ac82302"
server_key_version: "encryptionkey@0"
I20260928 10:54:35.179652 19995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "bef7295215fb4179818f49d684029947"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "fb7344a41595620147dde7e0ee4e4cc7"
server_key_iv: "8d2196114f3b8f49fa3f3a782ac82302"
server_key_version: "encryptionkey@0"
I20260928 10:54:35.187646 19995 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260928 10:54:35.192396 20010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:35.194312 19995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:54:35.194556 19995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "bef7295215fb4179818f49d684029947"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "fb7344a41595620147dde7e0ee4e4cc7"
server_key_iv: "8d2196114f3b8f49fa3f3a782ac82302"
server_key_version: "encryptionkey@0"
I20260928 10:54:35.194834 19995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:35.212730 19995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:35.218380 19995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:35.218799 19995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:35.220279 19995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:35.222229 19995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:35.222357 19995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:35.222512 19995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:35.222589 19995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:35.264400 19995 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:45513
I20260928 10:54:35.264439 20122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:45513 every 8 connection(s)
I20260928 10:54:35.265898 19995 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:54:35.275637 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 19995
I20260928 10:54:35.275892 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:54:35.276731 6030 external_mini_cluster.cc:1428] Setting key d1596e8e3fbf482b6df7cdcac46466ed
Sep 28 10:54:35 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:35.291224 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:35.269289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:53249 (local address 127.5.227.190:44379)
0928 10:54:35.269792 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:35.269798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:35.270945 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 10:54:35.272944 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:35.272949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:35.273044 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:35.273251 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:35.275207 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.276458 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.278516 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.278927 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.279094 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:35.283396 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:35.283402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:35.283405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:35.283445 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:35.286765 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:35.288126 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:35.288131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:35.288133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:35.288271 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:35.288854 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:35.288858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:35.288860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:35.289295 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:35.289430 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:35.290673 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0928 10:54:35.290885 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":259,"threads_started":1}
I20260928 10:54:35.292795 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:35.269879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:53249)
0928 10:54:35.270752 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:35.270805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:35.272637 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:35.273609 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:35.273619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:35.274206 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:35.274826 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:35.274845 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.276700 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.276704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:35.278323 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:35.278340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.279209 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.279218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:35.279480 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:35.280332 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:35.280357 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:35.283107 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:35.286970 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:35.286985 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:35.287009 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:35.287930 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:35.288397 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:35.288407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:35.288414 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:35.288711 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:35.289449 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:35.289462 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:35.290111 (+ 649us) client_negotiation.cc:769] Sending connection context
0928 10:54:35.290501 (+ 390us) client_negotiation.cc:240] Negotiation successful
0928 10:54:35.291175 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":314,"threads_started":1}
I20260928 10:54:35.294806 20123 heartbeater.cc:347] Connected to a master server at 127.5.227.190:44379
I20260928 10:54:35.295176 20123 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:35.296139 20123 heartbeater.cc:511] Master 127.5.227.190:44379 requested a full tablet report, sending...
I20260928 10:54:35.299108 19934 ts_manager.cc:194] Registered new tserver with Master: bef7295215fb4179818f49d684029947 (127.5.227.129:45513)
I20260928 10:54:35.301010 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:53249
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:54:35.340737 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:44379
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:35.717811 20129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:35.718129 20129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:35.718305 20129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:35.726681 20129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:35.726814 20129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:35.726991 20129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:54:35.739588 20129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:35.741132 20129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:35.742573 20129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:35.761529 20134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:35.761556 20135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:35.761610 20137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:35.763499 20129 server_base.cc:1034] running on GCE node
I20260928 10:54:35.764402 20129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:35.765861 20129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:35.767333 20129 hybrid_clock.cc:648] HybridClock initialized: now 1790592875767221 us; error 106 us; skew 500 ppm
I20260928 10:54:35.767678 20129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:35 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:35.772671 20129 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:54:35.774549 20129 webserver.cc:460] Webserver started at http://127.5.227.130:38809/ using document root <none> and password file <none>
I20260928 10:54:35.775386 20129 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:35.775781 20129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:35.776201 20129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:35.779845 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "79f81d0b710d42a5b2f0f62aacb3f403"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "9ed12373ca3a72a8b3c95c70a54cacb1"
server_key_iv: "e988b69ad5071eb3b75bb268f59bbf86"
server_key_version: "encryptionkey@0"
I20260928 10:54:35.780810 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "79f81d0b710d42a5b2f0f62aacb3f403"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "9ed12373ca3a72a8b3c95c70a54cacb1"
server_key_iv: "e988b69ad5071eb3b75bb268f59bbf86"
server_key_version: "encryptionkey@0"
I20260928 10:54:35.789295 20129 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260928 10:54:35.795135 20144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:35.796983 20129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:54:35.797394 20129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "79f81d0b710d42a5b2f0f62aacb3f403"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "9ed12373ca3a72a8b3c95c70a54cacb1"
server_key_iv: "e988b69ad5071eb3b75bb268f59bbf86"
server_key_version: "encryptionkey@0"
I20260928 10:54:35.797739 20129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:35.839545 20129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:35.845297 20129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:35.845682 20129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:35.846835 20129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:35.848692 20129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:35.848819 20129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:35.849000 20129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:35.849115 20129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:35.892210 20129 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:43755
I20260928 10:54:35.892346 20256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:43755 every 8 connection(s)
I20260928 10:54:35.893805 20129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:54:35.899770 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 20129
I20260928 10:54:35.900004 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:54:35.900656 6030 external_mini_cluster.cc:1428] Setting key b4fb0959e010588299e3765a8f66869b
Sep 28 10:54:35 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:35.922832 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:35.897595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:56185 (local address 127.5.227.190:44379)
0928 10:54:35.898162 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:35.898167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:35.899451 (+ 1284us) server_negotiation.cc:407] Connection header received
0928 10:54:35.902296 (+ 2845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:35.902302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:35.902412 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:35.902545 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:35.905194 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.906257 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.909249 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.910079 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.910279 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:35.915363 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:35.915369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:35.915372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:35.915415 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:35.918949 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:35.920028 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:35.920033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:35.920035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:35.920183 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:35.920710 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:35.920714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:35.920716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:35.921035 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:35.921211 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:35.922035 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 10:54:35.922307 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
I20260928 10:54:35.924119 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:35.898355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:56185)
0928 10:54:35.899219 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:35.899278 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:35.901962 (+ 2684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:35.902887 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:35.902901 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:35.903813 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:35.904849 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:35.904883 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.906508 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.906516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:35.909016 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:35.909040 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:35.910442 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:35.910455 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:35.910944 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:35.911932 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:35.911971 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:35.915009 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:35.919191 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:35.919209 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:35.919226 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:35.919814 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:35.920325 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:35.920331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:35.920334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:35.920565 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:35.921184 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:35.921193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:35.921608 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 10:54:35.921901 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 10:54:35.922653 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":329,"threads_started":1}
I20260928 10:54:35.925854 20257 heartbeater.cc:347] Connected to a master server at 127.5.227.190:44379
I20260928 10:54:35.926204 20257 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:35.926999 20257 heartbeater.cc:511] Master 127.5.227.190:44379 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:54:35.928791 19934 ts_manager.cc:194] Registered new tserver with Master: 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755)
I20260928 10:54:35.929896 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:56185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:54:35.965274 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:44379
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:36.304924 20123 heartbeater.cc:503] Master 127.5.227.190:44379 was elected leader, sending a full tablet report...
W20260928 10:54:36.364413 20263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:36.364763 20263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:36.364943 20263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:36.373129 20263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:36.373268 20263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:36.373448 20263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:54:36.384644 20263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:36.386089 20263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:36.387537 20263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:36.403946 20271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:36.403925 20269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:36.403995 20268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:36.410014 20263 server_base.cc:1034] running on GCE node
I20260928 10:54:36.410836 20263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:36.412400 20263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:36.413640 20263 hybrid_clock.cc:648] HybridClock initialized: now 1790592876413569 us; error 60 us; skew 500 ppm
I20260928 10:54:36.413962 20263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:36 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592876, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:36.417675 20263 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:54:36.419644 20263 webserver.cc:460] Webserver started at http://127.5.227.131:44121/ using document root <none> and password file <none>
I20260928 10:54:36.420526 20263 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:36.420684 20263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:36.421049 20263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:54:36.424664 20263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "c9273a6b555844ec9ca66403535ccb6a"
format_stamp: "Formatted at 2026-09-28 10:54:36 on dist-test-slave-8w8g"
server_key: "d4a1af75c95fd7471e91fab675973489"
server_key_iv: "1c809de53d135da1be2d39f0b94f9c39"
server_key_version: "encryptionkey@0"
I20260928 10:54:36.425561 20263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "c9273a6b555844ec9ca66403535ccb6a"
format_stamp: "Formatted at 2026-09-28 10:54:36 on dist-test-slave-8w8g"
server_key: "d4a1af75c95fd7471e91fab675973489"
server_key_iv: "1c809de53d135da1be2d39f0b94f9c39"
server_key_version: "encryptionkey@0"
I20260928 10:54:36.431958 20263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:54:36.436710 20278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:36.438943 20263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:54:36.439249 20263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "c9273a6b555844ec9ca66403535ccb6a"
format_stamp: "Formatted at 2026-09-28 10:54:36 on dist-test-slave-8w8g"
server_key: "d4a1af75c95fd7471e91fab675973489"
server_key_iv: "1c809de53d135da1be2d39f0b94f9c39"
server_key_version: "encryptionkey@0"
I20260928 10:54:36.439545 20263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:36.455713 20263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:36.460685 20263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:36.461006 20263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:36.462055 20263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:36.463972 20263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:36.464097 20263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:36.464274 20263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:36.464370 20263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:36.506357 20263 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:40729
I20260928 10:54:36.506407 20390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:40729 every 8 connection(s)
I20260928 10:54:36.507907 20263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:54:36.515492 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 20263
I20260928 10:54:36.515779 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:54:36.516613 6030 external_mini_cluster.cc:1428] Setting key fe8b855fe375fd6d34bbd09c5fbd1ea3
Sep 28 10:54:36 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592876, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:36.534694 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.511179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:60135 (local address 127.5.227.190:44379)
0928 10:54:36.511714 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:36.511719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:36.512909 (+ 1190us) server_negotiation.cc:407] Connection header received
0928 10:54:36.515017 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:36.515023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:36.515139 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:36.515347 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:36.517353 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.518317 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.520677 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.521224 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.521405 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.525575 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:36.525581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:36.525584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:36.525625 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:36.529011 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:36.530008 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:36.530015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:36.530019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:36.530175 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:36.530837 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:36.530843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:36.530846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:36.531390 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:36.531581 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:36.534182 (+ 2601us) server_negotiation.cc:299] Negotiation successful
0928 10:54:36.534396 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":237,"threads_started":1}
I20260928 10:54:36.535142 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.511861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:60135)
0928 10:54:36.512703 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:36.512750 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:36.514682 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:36.515634 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:36.515645 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:36.516390 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:36.517130 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.517150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.518556 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.518563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.520310 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.520326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.521518 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.521527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.521796 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.522586 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:36.522610 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:36.525246 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:36.529235 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:36.529251 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:36.529274 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:36.529827 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:36.530322 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:36.530328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:36.530332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:36.530649 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:36.531682 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:36.531691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:36.532283 (+ 592us) client_negotiation.cc:769] Sending connection context
0928 10:54:36.532688 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 10:54:36.533452 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":301,"threads_started":1}
I20260928 10:54:36.536975 20391 heartbeater.cc:347] Connected to a master server at 127.5.227.190:44379
I20260928 10:54:36.537282 20391 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:36.538039 20391 heartbeater.cc:511] Master 127.5.227.190:44379 requested a full tablet report, sending...
I20260928 10:54:36.539613 19934 ts_manager.cc:194] Registered new tserver with Master: c9273a6b555844ec9ca66403535ccb6a (127.5.227.131:40729)
I20260928 10:54:36.540518 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:60135
I20260928 10:54:36.546433 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:54:36.568781 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.551455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.5.227.190:44379)
0928 10:54:36.551684 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:36.551692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:36.552417 (+ 725us) server_negotiation.cc:407] Connection header received
0928 10:54:36.552567 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:36.552571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:36.552674 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:36.552768 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:36.554999 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.556048 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.558213 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.558723 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.558851 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.562255 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:36.562266 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:36.562269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:36.562312 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:36.565374 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:36.566257 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:36.566262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:36.566265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:36.566415 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:36.566951 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:36.566956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:36.566958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:36.567365 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:36.567579 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:36.568132 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 10:54:36.568351 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:54:36.573884 19934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:54:36.581316 19934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:54:36.610864 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.597294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.5.227.129:45513)
0928 10:54:36.602287 (+ 4993us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:36.602294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:36.602319 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:54:36.602401 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:36.602407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:36.602631 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:36.602857 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:36.603525 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.605283 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.607999 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.609526 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.609692 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.610006 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:36.610278 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 10:54:36.610455 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4841,"thread_start_us":4682,"threads_started":1}
I20260928 10:54:36.610976 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.596263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:40729 (local address 127.0.0.1:54734)
0928 10:54:36.598053 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:36.598113 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:36.598333 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:36.600794 (+ 2461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:36.600805 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:36.600846 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:36.601604 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.601623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.604845 (+ 3222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.604852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.607520 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.607536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.609891 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.609899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.610080 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.610119 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:54:36.610194 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 10:54:36.610431 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":666,"threads_started":1}
I20260928 10:54:36.611506 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.596677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:56056)
0928 10:54:36.599103 (+ 2426us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:36.599126 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:36.599257 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:36.602846 (+ 3589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:36.602851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:36.602880 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:36.603281 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.603295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.605438 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.605446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.607800 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.607813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.609745 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.609752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.609989 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.610063 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 10:54:36.610229 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 10:54:36.610449 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"mutex_wait_us":6,"thread_start_us":2530,"threads_started":1}
I20260928 10:54:36.611905 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.596582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.5.227.131:40729)
0928 10:54:36.599932 (+ 3350us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:36.599942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:36.599976 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:54:36.600110 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:36.600121 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:36.600513 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:36.601942 (+ 1429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:36.602297 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.604646 (+ 2349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.607710 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.609603 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.611021 (+ 1418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.611265 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:36.611344 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 10:54:36.611498 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3063,"thread_start_us":4109,"threads_started":1}
I20260928 10:54:36.614163 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.599516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43755 (local address 127.0.0.1:57000)
0928 10:54:36.601644 (+ 2128us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:36.601662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:36.601774 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:36.605783 (+ 4009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:36.605789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:36.605831 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:36.606481 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.606495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.608658 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.608664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.611639 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.611657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.613196 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.613204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.613395 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.613431 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:54:36.613544 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 10:54:36.613717 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1983,"mutex_wait_us":29,"thread_start_us":1809,"threads_started":1}
I20260928 10:54:36.615099 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.599864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.5.227.130:43755)
0928 10:54:36.605069 (+ 5205us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:36.605079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:36.605121 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:54:36.605238 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:36.605247 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:36.605544 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:36.605803 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:36.606689 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.608446 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.611872 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.612935 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.613106 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.613293 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:36.614549 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0928 10:54:36.614748 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5034,"thread_start_us":4913,"threads_started":1}
I20260928 10:54:36.616070 20326 tablet_service.cc:1467] Processing CreateTablet for tablet 1713efad1df749028d0e7fdf1383ba1a (DEFAULT_TABLE table=test-table [id=c616ec2c4d1e44249063c767373ce8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:36.616091 20058 tablet_service.cc:1467] Processing CreateTablet for tablet 1713efad1df749028d0e7fdf1383ba1a (DEFAULT_TABLE table=test-table [id=c616ec2c4d1e44249063c767373ce8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:36.617466 20058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1713efad1df749028d0e7fdf1383ba1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:36.617486 20326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1713efad1df749028d0e7fdf1383ba1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:36.620841 20192 tablet_service.cc:1467] Processing CreateTablet for tablet 1713efad1df749028d0e7fdf1383ba1a (DEFAULT_TABLE table=test-table [id=c616ec2c4d1e44249063c767373ce8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:36.622468 20192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1713efad1df749028d0e7fdf1383ba1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:36.629702 20410 tablet_bootstrap.cc:492] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Bootstrap starting.
I20260928 10:54:36.633033 20410 tablet_bootstrap.cc:654] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:36.634130 20410 log.cc:824] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:36.635612 20412 tablet_bootstrap.cc:492] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Bootstrap starting.
I20260928 10:54:36.637665 20410 tablet_bootstrap.cc:492] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: No bootstrap required, opened a new log
I20260928 10:54:36.637948 20410 ts_tablet_manager.cc:1397] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 10:54:36.640282 20413 tablet_bootstrap.cc:492] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Bootstrap starting.
I20260928 10:54:36.640684 20412 tablet_bootstrap.cc:654] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:36.642230 20412 log.cc:824] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:36.644457 20413 tablet_bootstrap.cc:654] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:36.645733 20413 log.cc:824] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:36.646260 20412 tablet_bootstrap.cc:492] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: No bootstrap required, opened a new log
I20260928 10:54:36.646703 20412 ts_tablet_manager.cc:1397] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 10:54:36.648854 20410 raft_consensus.cc:348] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.649353 20410 raft_consensus.cc:374] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:36.649464 20410 raft_consensus.cc:729] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9273a6b555844ec9ca66403535ccb6a, State: Initialized, Role: FOLLOWER
I20260928 10:54:36.649605 20413 tablet_bootstrap.cc:492] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: No bootstrap required, opened a new log
I20260928 10:54:36.649907 20413 ts_tablet_manager.cc:1397] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 10:54:36.650146 20410 consensus_queue.cc:260] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.651343 20391 heartbeater.cc:503] Master 127.5.227.190:44379 was elected leader, sending a full tablet report...
I20260928 10:54:36.652034 20410 ts_tablet_manager.cc:1428] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260928 10:54:36.657426 20412 raft_consensus.cc:348] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.657919 20412 raft_consensus.cc:374] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:36.658053 20412 raft_consensus.cc:729] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef7295215fb4179818f49d684029947, State: Initialized, Role: FOLLOWER
I20260928 10:54:36.658725 20412 consensus_queue.cc:260] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.660642 20412 ts_tablet_manager.cc:1428] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 10:54:36.661408 20413 raft_consensus.cc:348] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.662002 20413 raft_consensus.cc:374] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:36.662108 20413 raft_consensus.cc:729] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f81d0b710d42a5b2f0f62aacb3f403, State: Initialized, Role: FOLLOWER
I20260928 10:54:36.662726 20413 consensus_queue.cc:260] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.664193 20257 heartbeater.cc:503] Master 127.5.227.190:44379 was elected leader, sending a full tablet report...
I20260928 10:54:36.665052 20413 ts_tablet_manager.cc:1428] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260928 10:54:36.759843 20392 tablet.cc:2369] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:54:36.769178 20124 tablet.cc:2369] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:54:36.896962 20258 tablet.cc:2369] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:54:36.972546 20416 raft_consensus.cc:482] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:54:36.972865 20416 raft_consensus.cc:504] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.974731 20416 leader_election.cc:288] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bef7295215fb4179818f49d684029947 (127.5.227.129:45513), 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755)
I20260928 10:54:36.982589 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.975285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:56403 (local address 127.5.227.129:45513)
0928 10:54:36.975567 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:36.975573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:36.975626 (+ 53us) server_negotiation.cc:407] Connection header received
0928 10:54:36.975891 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:36.975897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:36.975992 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:36.976153 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:36.976718 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.977759 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.980172 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.981455 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.981709 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.981776 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:36.982244 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 10:54:36.982368 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 10:54:36.982630 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.975267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.5.227.131:56403)
0928 10:54:36.975489 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:36.975508 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:36.975661 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:36.976198 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:36.976204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:36.976237 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:36.976526 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.976540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.977880 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.977888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.979896 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.979909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.981710 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.981726 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.981931 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.982017 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 10:54:36.982148 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 10:54:36.982300 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":1}
I20260928 10:54:36.984169 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.976008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:35419 (local address 127.5.227.130:43755)
0928 10:54:36.976380 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:36.976390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:36.976822 (+ 432us) server_negotiation.cc:407] Connection header received
0928 10:54:36.977035 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:36.977039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:36.977105 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:36.977230 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:36.977811 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.979051 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.981703 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.982988 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.983168 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.983248 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:36.983811 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 10:54:36.983955 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260928 10:54:36.983906 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713efad1df749028d0e7fdf1383ba1a" candidate_uuid: "c9273a6b555844ec9ca66403535ccb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef7295215fb4179818f49d684029947" is_pre_election: true
I20260928 10:54:36.984454 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:36.975851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43755 (local address 127.5.227.131:35419)
0928 10:54:36.976700 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:36.976726 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:36.976871 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:36.977258 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:36.977262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:36.977289 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:36.977636 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.977646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.979254 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.979259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.981507 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:36.981525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:36.983307 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:36.983314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:36.983522 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:36.983562 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:54:36.983684 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 10:54:36.983868 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":570,"threads_started":1}
I20260928 10:54:36.984843 20078 raft_consensus.cc:2454] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9273a6b555844ec9ca66403535ccb6a in term 0.
I20260928 10:54:36.985704 20212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713efad1df749028d0e7fdf1383ba1a" candidate_uuid: "c9273a6b555844ec9ca66403535ccb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" is_pre_election: true
I20260928 10:54:36.985867 20279 leader_election.cc:302] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef7295215fb4179818f49d684029947, c9273a6b555844ec9ca66403535ccb6a; no voters:
I20260928 10:54:36.986364 20416 raft_consensus.cc:2781] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:54:36.986368 20212 raft_consensus.cc:2454] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9273a6b555844ec9ca66403535ccb6a in term 0.
I20260928 10:54:36.986501 20416 raft_consensus.cc:482] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:54:36.986640 20416 raft_consensus.cc:3037] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:36.989850 20416 raft_consensus.cc:504] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:36.990641 20416 leader_election.cc:288] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [CANDIDATE]: Term 1 election: Requested vote from peers bef7295215fb4179818f49d684029947 (127.5.227.129:45513), 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755)
I20260928 10:54:36.991343 20212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713efad1df749028d0e7fdf1383ba1a" candidate_uuid: "c9273a6b555844ec9ca66403535ccb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f81d0b710d42a5b2f0f62aacb3f403"
I20260928 10:54:36.991477 20078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713efad1df749028d0e7fdf1383ba1a" candidate_uuid: "c9273a6b555844ec9ca66403535ccb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef7295215fb4179818f49d684029947"
I20260928 10:54:36.991598 20212 raft_consensus.cc:3037] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:36.991920 20078 raft_consensus.cc:3037] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:36.994414 20078 raft_consensus.cc:2454] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9273a6b555844ec9ca66403535ccb6a in term 1.
I20260928 10:54:36.995128 20212 raft_consensus.cc:2454] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9273a6b555844ec9ca66403535ccb6a in term 1.
I20260928 10:54:36.995469 20279 leader_election.cc:302] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef7295215fb4179818f49d684029947, c9273a6b555844ec9ca66403535ccb6a; no voters:
I20260928 10:54:36.996407 20416 raft_consensus.cc:2781] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:54:36.997181 20416 raft_consensus.cc:686] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 1 LEADER]: Becoming Leader. State: Replica: c9273a6b555844ec9ca66403535ccb6a, State: Running, Role: LEADER
I20260928 10:54:36.998240 20416 consensus_queue.cc:237] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:37.007148 19933 catalog_manager.cc:5495] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c9273a6b555844ec9ca66403535ccb6a (127.5.227.131). New cstate: current_term: 1 leader_uuid: "c9273a6b555844ec9ca66403535ccb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } health_report { overall_health: UNKNOWN } } }
I20260928 10:54:37.072836 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:37.065657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.5.227.131:40729)
0928 10:54:37.065897 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:37.065902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:37.065953 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:54:37.066384 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:37.066388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:37.066482 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:37.066688 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:54:37.067072 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.067948 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.069801 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.070228 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.070348 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:37.071038 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:54:37.072050 (+ 1012us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:54:37.072205 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:37.072499 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 10:54:37.072586 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 10:54:37.084141 20212 raft_consensus.cc:1261] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Refusing update from remote peer c9273a6b555844ec9ca66403535ccb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:37.084363 20078 raft_consensus.cc:1261] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 1 FOLLOWER]: Refusing update from remote peer c9273a6b555844ec9ca66403535ccb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:37.085466 20420 consensus_queue.cc:1035] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:37.086207 20416 consensus_queue.cc:1035] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:37.123569 19934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:54:37.123960 19934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:54:37.128731 19934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Sending DeleteTablet for 3 replicas of tablet 1713efad1df749028d0e7fdf1383ba1a
I20260928 10:54:37.130854 20326 tablet_service.cc:1514] Processing DeleteTablet for tablet 1713efad1df749028d0e7fdf1383ba1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:37 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:54734
I20260928 10:54:37.130854 20192 tablet_service.cc:1514] Processing DeleteTablet for tablet 1713efad1df749028d0e7fdf1383ba1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:37 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:57000
I20260928 10:54:37.130849 20058 tablet_service.cc:1514] Processing DeleteTablet for tablet 1713efad1df749028d0e7fdf1383ba1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:37 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:56056
I20260928 10:54:37.132341 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19995
I20260928 10:54:37.147703 20435 tablet_replica.cc:330] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: stopping tablet replica
I20260928 10:54:37.148617 20435 raft_consensus.cc:2229] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:54:37.149382 20435 raft_consensus.cc:2258] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:37.150339 20434 tablet_replica.cc:330] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: stopping tablet replica
I20260928 10:54:37.151152 20434 raft_consensus.cc:2229] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:54:37.152084 20435 ts_tablet_manager.cc:1905] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:37.156062 20433 tablet_replica.cc:330] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: stopping tablet replica
I20260928 10:54:37.152720 20434 raft_consensus.cc:2258] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:37.156622 20433 raft_consensus.cc:2229] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:54:37.157212 20433 raft_consensus.cc:2258] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:37.158660 20434 ts_tablet_manager.cc:1905] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:37.160008 20433 ts_tablet_manager.cc:1905] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:37.160660 20435 ts_tablet_manager.cc:1918] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:37.160853 20435 log.cc:1196] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/1713efad1df749028d0e7fdf1383ba1a
I20260928 10:54:37.161433 20435 ts_tablet_manager.cc:1939] T 1713efad1df749028d0e7fdf1383ba1a P 79f81d0b710d42a5b2f0f62aacb3f403: Deleting consensus metadata
I20260928 10:54:37.163981 19918 catalog_manager.cc:4841] TS 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755): tablet 1713efad1df749028d0e7fdf1383ba1a (table test-table [id=c616ec2c4d1e44249063c767373ce8db]) successfully deleted
Sep 28 10:54:37 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:54:37.169162 20434 ts_tablet_manager.cc:1918] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:37.169164 20433 ts_tablet_manager.cc:1918] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:37.169379 20434 log.cc:1196] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/1713efad1df749028d0e7fdf1383ba1a
I20260928 10:54:37.169380 20433 log.cc:1196] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/1713efad1df749028d0e7fdf1383ba1a
I20260928 10:54:37.170228 20433 ts_tablet_manager.cc:1939] T 1713efad1df749028d0e7fdf1383ba1a P bef7295215fb4179818f49d684029947: Deleting consensus metadata
I20260928 10:54:37.170225 20434 ts_tablet_manager.cc:1939] T 1713efad1df749028d0e7fdf1383ba1a P c9273a6b555844ec9ca66403535ccb6a: Deleting consensus metadata
I20260928 10:54:37.175987 19919 catalog_manager.cc:4841] TS c9273a6b555844ec9ca66403535ccb6a (127.5.227.131:40729): tablet 1713efad1df749028d0e7fdf1383ba1a (table test-table [id=c616ec2c4d1e44249063c767373ce8db]) successfully deleted
I20260928 10:54:37.177166 19918 catalog_manager.cc:4841] TS bef7295215fb4179818f49d684029947 (127.5.227.129:45513): tablet 1713efad1df749028d0e7fdf1383ba1a (table test-table [id=c616ec2c4d1e44249063c767373ce8db]) successfully deleted
I20260928 10:54:37.177481 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:37.133756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.5.227.129:45513)
0928 10:54:37.133962 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:37.133969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:37.140534 (+ 6565us) server_negotiation.cc:407] Connection header received
0928 10:54:37.142458 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:37.142469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:37.142598 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:37.142753 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:37.152111 (+ 9358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.155430 (+ 3319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.158787 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.159862 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.160012 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:37.167528 (+ 7516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:37.167549 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:37.167570 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:37.167628 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:37.172499 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:37.173495 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:37.173507 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:37.173511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:37.173718 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:37.174466 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:37.174475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:37.174480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:37.175322 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:37.175702 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:37.176756 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0928 10:54:37.177082 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:54:37.178828 20118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:37.313189 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19995
I20260928 10:54:37.332408 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 20129
Sep 28 10:54:37 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:54:37.352589 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:37.333410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.5.227.130:43755)
0928 10:54:37.333665 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:37.333673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:37.333695 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:54:37.333914 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:37.333920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:37.333997 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:37.334126 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:37.335447 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.336427 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.337876 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.338286 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.338431 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:37.343810 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:37.343833 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:37.343856 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:37.343916 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:37.348446 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:37.349802 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:37.349813 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:37.349817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:37.350030 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:37.350526 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:37.350531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:37.350535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:37.351037 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:37.351377 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:37.351842 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 10:54:37.352098 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:54:37.353483 20252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:37.451156 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 20129
I20260928 10:54:37.477035 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 20263
Sep 28 10:54:37 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:54:37.496552 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:37.477851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.5.227.131:40729)
0928 10:54:37.478078 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:37.478084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:37.478109 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:54:37.478300 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:37.478308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:37.478417 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:37.478562 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:37.480433 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.481671 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.483348 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:37.483920 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:37.484106 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:37.489333 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:37.489348 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:37.489364 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:37.489408 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:37.493150 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:37.494045 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:37.494054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:37.494057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:37.494244 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:37.494687 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:37.494693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:37.494696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:37.495066 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:37.495476 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:37.495806 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 10:54:37.496058 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 10:54:37.497624 20386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:37.636309 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 20263
I20260928 10:54:37.658427 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 19902
I20260928 10:54:37.659917 19964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:37.779678 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 19902
I20260928 10:54:37.802353 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44379
--webserver_interface=127.5.227.190
--webserver_port=36719
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:38.169720 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:38.170117 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:38.170310 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:38.179194 20441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:38.179344 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:38.179455 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:38.179585 20441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:38.179670 20441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:38.192862 20441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:38.194736 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:38.196440 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:38.226285 20447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:38.226051 20448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:38.226081 20450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:38.227676 20441 server_base.cc:1034] running on GCE node
I20260928 10:54:38.228886 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:38.230962 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:38.232429 20441 hybrid_clock.cc:648] HybridClock initialized: now 1790592878232359 us; error 58 us; skew 500 ppm
I20260928 10:54:38.232933 20441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:38 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592878, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:38.238384 20441 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:54:38.241077 20441 webserver.cc:460] Webserver started at http://127.5.227.190:36719/ using document root <none> and password file <none>
I20260928 10:54:38.242008 20441 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:38.242205 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:38.252411 20441 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:54:38.256771 20457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:38.259109 20441 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:54:38.259524 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "39346fd2b3bc46b2a5fc38bfebc979de"
format_stamp: "Formatted at 2026-09-28 10:54:34 on dist-test-slave-8w8g"
server_key: "f54af56296eb2a8a71d240a12704a731"
server_key_iv: "45f9877e5bf35b495d3aeb6b9bbfe854"
server_key_version: "encryptionkey@0"
I20260928 10:54:38.261373 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:38.296452 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:38.303282 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:38.303659 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:38.327592 20441 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:44379
I20260928 10:54:38.327610 20508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:44379 every 8 connection(s)
I20260928 10:54:38.329367 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:54:38.331673 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 20441
I20260928 10:54:38.332644 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:45513
--local_ip_for_outbound_sockets=127.5.227.129
--tserver_master_addrs=127.5.227.190:44379
--webserver_port=33421
--webserver_interface=127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:38.337831 20509 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:54:38.342154 20509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Bootstrap starting.
I20260928 10:54:38.372189 20509 log.cc:824] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:38.387203 20509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:54:38.387930 20509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Bootstrap complete.
I20260928 10:54:38.399008 20509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:38.400110 20509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39346fd2b3bc46b2a5fc38bfebc979de, State: Initialized, Role: FOLLOWER
I20260928 10:54:38.400873 20509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:38.401172 20509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:54:38.401322 20509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:54:38.401525 20509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:54:38.406199 20509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:38.406985 20509 leader_election.cc:302] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39346fd2b3bc46b2a5fc38bfebc979de; no voters:
I20260928 10:54:38.407804 20509 leader_election.cc:288] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:54:38.407984 20513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:54:38.410020 20513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 LEADER]: Becoming Leader. State: Replica: 39346fd2b3bc46b2a5fc38bfebc979de, State: Running, Role: LEADER
I20260928 10:54:38.411203 20513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:38.411583 20509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:38.414192 20514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } } }
I20260928 10:54:38.414654 20514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:38.416759 20515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39346fd2b3bc46b2a5fc38bfebc979de. Latest consensus state: current_term: 2 leader_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } } }
I20260928 10:54:38.417095 20515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:38.422242 20518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:54:38.434547 20518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:54:38.437603 20518 catalog_manager.cc:1259] Loaded cluster ID: 2820cc864c0f424aa06e8f2dfd9ffac7
I20260928 10:54:38.437788 20518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:54:38.442747 20518 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 10:54:38.470877 20518 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 10:54:38.485785 20518 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:54:38.489125 20518 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Loaded TSK: 0
I20260928 10:54:38.491225 20518 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:54:38.773373 20511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:38.773707 20511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:38.773859 20511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:38.781574 20511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:38.781708 20511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:38.781886 20511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:54:38.793433 20511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:45513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=33421
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:38.794909 20511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:38.796695 20511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:38.814723 20538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:38.815629 20540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:38.814803 20537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:38.822440 20511 server_base.cc:1034] running on GCE node
I20260928 10:54:38.823585 20511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:38.825129 20511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:38.826442 20511 hybrid_clock.cc:648] HybridClock initialized: now 1790592878826379 us; error 63 us; skew 500 ppm
I20260928 10:54:38.826884 20511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:38 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592878, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:38.831247 20511 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:54:38.833374 20511 webserver.cc:460] Webserver started at http://127.5.227.129:33421/ using document root <none> and password file <none>
I20260928 10:54:38.834419 20511 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:38.834594 20511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:38.842855 20511 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:54:38.847323 20547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:38.849413 20511 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:54:38.849633 20511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "bef7295215fb4179818f49d684029947"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "fb7344a41595620147dde7e0ee4e4cc7"
server_key_iv: "8d2196114f3b8f49fa3f3a782ac82302"
server_key_version: "encryptionkey@0"
I20260928 10:54:38.851181 20511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:38.877079 20511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:38.882899 20511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:38.883235 20511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:38.884604 20511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:38.886538 20511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:38.886651 20511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:38.886750 20511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:38.886821 20511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:38.928171 20511 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:45513
I20260928 10:54:38.928192 20659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:45513 every 8 connection(s)
I20260928 10:54:38.929838 20511 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:54:38.935885 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 20511
I20260928 10:54:38.937049 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:43755
--local_ip_for_outbound_sockets=127.5.227.130
--tserver_master_addrs=127.5.227.190:44379
--webserver_port=38809
--webserver_interface=127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:54:38 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592878, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:38.963338 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:38.933734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:41979)
0928 10:54:38.934768 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:38.934808 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:38.938284 (+ 3476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:38.940333 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:38.940347 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:38.941313 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:38.942333 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:38.942369 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:38.944681 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:38.944686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:38.946061 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:38.946078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:38.948181 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:38.948189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:38.948418 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:38.949320 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:38.949346 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:38.952074 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:38.957907 (+ 5833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:38.957919 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:38.957936 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:38.958470 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:38.959384 (+ 914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:38.959389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:38.959391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:38.959879 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:38.961175 (+ 1296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:38.961182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:38.961578 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 10:54:38.961840 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 10:54:38.962281 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":509,"threads_started":1}
I20260928 10:54:38.964159 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:38.933410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:41979 (local address 127.5.227.190:44379)
0928 10:54:38.934123 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:38.934136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:38.934990 (+ 854us) server_negotiation.cc:407] Connection header received
0928 10:54:38.939112 (+ 4122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:38.939164 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:38.939617 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:38.940073 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:38.942814 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:38.944414 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:38.946266 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:38.947808 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:38.948076 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:38.952493 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:38.952510 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:38.952536 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:38.952611 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:38.957638 (+ 5027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:38.958727 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:38.958742 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:38.958755 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:38.959171 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:38.960024 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:38.960033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:38.960038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:38.960915 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:38.961181 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:38.962174 (+ 993us) server_negotiation.cc:299] Negotiation successful
0928 10:54:38.962714 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":349,"threads_started":1}
I20260928 10:54:38.967356 20660 heartbeater.cc:347] Connected to a master server at 127.5.227.190:44379
I20260928 10:54:38.967787 20660 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:38.968858 20660 heartbeater.cc:511] Master 127.5.227.190:44379 requested a full tablet report, sending...
I20260928 10:54:38.971504 20474 ts_manager.cc:194] Registered new tserver with Master: bef7295215fb4179818f49d684029947 (127.5.227.129:45513)
I20260928 10:54:38.973248 20474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:41979
W20260928 10:54:39.325337 20664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:39.325923 20664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:39.326187 20664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:39.334985 20664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:39.335207 20664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:39.335407 20664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:54:39.347146 20664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:43755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=38809
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:39.348863 20664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:39.350679 20664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:39.368451 20672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:39.368389 20670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:39.368949 20669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:39.370714 20664 server_base.cc:1034] running on GCE node
I20260928 10:54:39.371508 20664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:39.372762 20664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:39.374053 20664 hybrid_clock.cc:648] HybridClock initialized: now 1790592879374017 us; error 81 us; skew 500 ppm
I20260928 10:54:39.374523 20664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:39 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:39.379254 20664 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:54:39.381206 20664 webserver.cc:460] Webserver started at http://127.5.227.130:38809/ using document root <none> and password file <none>
I20260928 10:54:39.381983 20664 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:39.382146 20664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:39.388850 20664 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 10:54:39.392483 20679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:39.394465 20664 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 10:54:39.394709 20664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "79f81d0b710d42a5b2f0f62aacb3f403"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "9ed12373ca3a72a8b3c95c70a54cacb1"
server_key_iv: "e988b69ad5071eb3b75bb268f59bbf86"
server_key_version: "encryptionkey@0"
I20260928 10:54:39.396162 20664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:39.416236 20664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:39.421898 20664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:39.422169 20664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:39.423267 20664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:39.424844 20664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:39.424957 20664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:39.425184 20664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:39.425314 20664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:39.470079 20664 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:43755
I20260928 10:54:39.470220 20791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:43755 every 8 connection(s)
I20260928 10:54:39.471640 20664 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:54:39.479884 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 20664
I20260928 10:54:39.480703 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:40729
--local_ip_for_outbound_sockets=127.5.227.131
--tserver_master_addrs=127.5.227.190:44379
--webserver_port=44121
--webserver_interface=127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:54:39 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:39.506604 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:39.474596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:44187 (local address 127.5.227.190:44379)
0928 10:54:39.475153 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:39.475158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:39.476203 (+ 1045us) server_negotiation.cc:407] Connection header received
0928 10:54:39.478278 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:39.478285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:39.478389 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:39.478525 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:39.480983 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:39.482876 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:39.486472 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:39.486956 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:39.487077 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:39.497677 (+ 10600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:39.497683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:39.497687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:39.497732 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:39.501874 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:39.503236 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:39.503242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:39.503244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:39.503428 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:39.504166 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:39.504172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:39.504174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:39.504526 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:39.504725 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:39.505913 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0928 10:54:39.506276 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":265,"threads_started":1}
I20260928 10:54:39.508174 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:39.475020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:44187)
0928 10:54:39.476005 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:39.476058 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:39.477952 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:39.478820 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:39.478830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:39.479637 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:39.480632 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:39.480668 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:39.483143 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:39.483153 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:39.486170 (+ 3017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:39.486208 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:39.487266 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:39.487278 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:39.487704 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:39.494046 (+ 6342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:39.494088 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:39.497283 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:39.502108 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:39.502121 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:39.502144 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:39.503009 (+ 865us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:39.503570 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:39.503578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:39.503585 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:39.503965 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:39.504704 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:39.504717 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:39.505287 (+ 570us) client_negotiation.cc:769] Sending connection context
0928 10:54:39.505722 (+ 435us) client_negotiation.cc:240] Negotiation successful
0928 10:54:39.506439 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":315,"threads_started":1}
I20260928 10:54:39.510001 20792 heartbeater.cc:347] Connected to a master server at 127.5.227.190:44379
I20260928 10:54:39.510392 20792 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:39.511270 20792 heartbeater.cc:511] Master 127.5.227.190:44379 requested a full tablet report, sending...
I20260928 10:54:39.513060 20474 ts_manager.cc:194] Registered new tserver with Master: 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755)
I20260928 10:54:39.514268 20474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:44187
W20260928 10:54:39.844821 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:39.845152 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:39.845257 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:39.853132 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:39.853268 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:39.853387 20796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:54:39.864048 20796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:40729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=44121
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:39.865408 20796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:39.866751 20796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:39.882306 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:39.882628 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:39.883050 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:39.883670 20796 server_base.cc:1034] running on GCE node
I20260928 10:54:39.884405 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:39.885543 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:39.886813 20796 hybrid_clock.cc:648] HybridClock initialized: now 1790592879886760 us; error 49 us; skew 500 ppm
I20260928 10:54:39.887245 20796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:39 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:39.890885 20796 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:54:39.892774 20796 webserver.cc:460] Webserver started at http://127.5.227.131:44121/ using document root <none> and password file <none>
I20260928 10:54:39.893534 20796 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:39.893713 20796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:39.899699 20796 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:54:39.903370 20811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:39.905481 20796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:54:39.905781 20796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "c9273a6b555844ec9ca66403535ccb6a"
format_stamp: "Formatted at 2026-09-28 10:54:36 on dist-test-slave-8w8g"
server_key: "d4a1af75c95fd7471e91fab675973489"
server_key_iv: "1c809de53d135da1be2d39f0b94f9c39"
server_key_version: "encryptionkey@0"
I20260928 10:54:39.906961 20796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:39.949805 20796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:39.955251 20796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:39.955597 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:39.957069 20796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:39.958834 20796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:39.958987 20796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:39.959161 20796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:39.959267 20796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:39.977670 20660 heartbeater.cc:503] Master 127.5.227.190:44379 was elected leader, sending a full tablet report...
I20260928 10:54:40.003403 20796 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:40729
I20260928 10:54:40.003446 20923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:40729 every 8 connection(s)
I20260928 10:54:40.005573 20796 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:54:40.010921 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 20796
Sep 28 10:54:40 dist-test-slave-8w8g krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592879, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:54:40.029222 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.012309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.5.227.190:44379)
0928 10:54:40.012989 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.012995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.013051 (+ 56us) server_negotiation.cc:407] Connection header received
0928 10:54:40.013244 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.013249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.013330 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.013513 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:40.015074 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.016225 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.017781 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.018383 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.018525 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.021674 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:40.021683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:40.021688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:40.021733 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:40.025868 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.026802 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.026807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.026810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.026955 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.027412 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.027419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.027422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.027826 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:40.028134 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.028462 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.028825 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":285,"threads_started":1}
I20260928 10:54:40.033685 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.008831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:58647 (local address 127.5.227.190:44379)
0928 10:54:40.009468 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.009476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.010753 (+ 1277us) server_negotiation.cc:407] Connection header received
0928 10:54:40.014381 (+ 3628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.014387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.014493 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.014619 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:40.016799 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.017580 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.019564 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.020085 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.020352 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.025353 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:40.025359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:40.025362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:40.025415 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:40.029186 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.030415 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.030420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.030422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.030550 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.031289 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.031297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.031300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.031822 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:40.032036 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.033030 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.033353 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":400,"threads_started":1}
I20260928 10:54:40.034585 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.009506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:58647)
0928 10:54:40.010529 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.010590 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:40.014083 (+ 3493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:40.014900 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:40.014909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:40.015773 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:54:40.016599 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.016621 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.017763 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.017769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.019405 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.019419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.020485 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.020496 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.020795 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.021767 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:54:40.021815 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:54:40.025037 (+ 3222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:54:40.029413 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:40.029426 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:40.029449 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:40.030219 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:40.030742 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:54:40.030748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:54:40.030752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:54:40.031107 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:54:40.031993 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:54:40.032029 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:54:40.032510 (+ 481us) client_negotiation.cc:769] Sending connection context
0928 10:54:40.032896 (+ 386us) client_negotiation.cc:240] Negotiation successful
0928 10:54:40.033368 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":327,"threads_started":1}
I20260928 10:54:40.036240 20924 heartbeater.cc:347] Connected to a master server at 127.5.227.190:44379
I20260928 10:54:40.036600 20924 heartbeater.cc:464] Registering TS with master...
I20260928 10:54:40.037426 20924 heartbeater.cc:511] Master 127.5.227.190:44379 requested a full tablet report, sending...
I20260928 10:54:40.039165 20474 ts_manager.cc:194] Registered new tserver with Master: c9273a6b555844ec9ca66403535ccb6a (127.5.227.131:40729)
I20260928 10:54:40.040161 20474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:58647
I20260928 10:54:40.046702 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:54:40.056205 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.047943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.5.227.190:44379)
0928 10:54:40.048189 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.048195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.048583 (+ 388us) server_negotiation.cc:407] Connection header received
0928 10:54:40.048841 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.048846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.049055 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.049256 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:54:40.049814 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.050648 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.052398 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.053167 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.053325 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.054290 (+ 965us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:54:40.055353 (+ 1063us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:54:40.055495 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.055830 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.055929 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 10:54:40.057750 20474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:54:40.066744 20474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:54:40.106374 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.081740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:56080)
0928 10:54:40.092800 (+ 11060us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.092850 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:40.093099 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:40.094099 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:40.094115 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:40.094170 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:40.094819 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.094839 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.098921 (+ 4082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.098931 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.102718 (+ 3787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.102739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.104928 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.104936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.105333 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.105407 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 10:54:40.105645 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 10:54:40.105910 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10779,"thread_start_us":10600,"threads_started":1}
I20260928 10:54:40.106375 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.081863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.5.227.129:45513)
0928 10:54:40.092221 (+ 10358us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.092235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.093340 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 10:54:40.093465 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.093474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.093780 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.095193 (+ 1413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:40.095544 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.098654 (+ 3110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.102988 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.104647 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.104808 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.105107 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.105666 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.105846 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10149,"thread_start_us":9973,"threads_started":1}
I20260928 10:54:40.112478 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.082009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:40729 (local address 127.0.0.1:54760)
0928 10:54:40.098959 (+ 16950us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.098979 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:40.099314 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:40.100779 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:40.100788 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:40.100837 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:40.101538 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.101555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.105776 (+ 4221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.105782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.108614 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.108633 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.111495 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.111502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.111695 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.111772 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 10:54:40.111883 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 10:54:40.112080 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16792,"thread_start_us":16631,"threads_started":1}
I20260928 10:54:40.112875 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.082617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.5.227.131:40729)
0928 10:54:40.098060 (+ 15443us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.098072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.099245 (+ 1173us) server_negotiation.cc:407] Connection header received
0928 10:54:40.099557 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.099570 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.099928 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.100318 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:40.103563 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.105570 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.108937 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.111204 (+ 2267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.111366 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.111630 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.112174 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.112356 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15196,"thread_start_us":15473,"threads_started":1}
I20260928 10:54:40.113183 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.093796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.5.227.130:43755)
0928 10:54:40.103043 (+ 9247us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.103054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.103585 (+ 531us) server_negotiation.cc:407] Connection header received
0928 10:54:40.103712 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.103724 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.104085 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.104334 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:40.105061 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.106904 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.109787 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.111376 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.111660 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.111974 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.112586 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.112769 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9046,"thread_start_us":8912,"threads_started":1}
I20260928 10:54:40.113097 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.099752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43755 (local address 127.0.0.1:57014)
0928 10:54:40.103413 (+ 3661us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.103434 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:40.103568 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:40.104332 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:40.104337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:40.104366 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:40.104783 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.104810 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.107080 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.107087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.109550 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.109568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.111872 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.111880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.112139 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.112190 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 10:54:40.112300 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 10:54:40.112454 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3515,"thread_start_us":3392,"threads_started":1}
I20260928 10:54:40.112195 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee8c650b5a04f888164405cc15e21c8 (DEFAULT_TABLE table=test-table [id=b385e002552e42028b77f4fafa713bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:40.114691 20595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee8c650b5a04f888164405cc15e21c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:40.117300 20727 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee8c650b5a04f888164405cc15e21c8 (DEFAULT_TABLE table=test-table [id=b385e002552e42028b77f4fafa713bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:40.117437 20859 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee8c650b5a04f888164405cc15e21c8 (DEFAULT_TABLE table=test-table [id=b385e002552e42028b77f4fafa713bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:54:40.118707 20727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee8c650b5a04f888164405cc15e21c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:40.118968 20859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee8c650b5a04f888164405cc15e21c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:54:40.131814 20941 tablet_bootstrap.cc:492] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Bootstrap starting.
I20260928 10:54:40.133941 20942 tablet_bootstrap.cc:492] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Bootstrap starting.
I20260928 10:54:40.136597 20941 tablet_bootstrap.cc:654] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:40.137907 20942 tablet_bootstrap.cc:654] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:40.138264 20941 log.cc:824] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:40.138767 20943 tablet_bootstrap.cc:492] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Bootstrap starting.
I20260928 10:54:40.139356 20942 log.cc:824] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:40.142967 20942 tablet_bootstrap.cc:492] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: No bootstrap required, opened a new log
I20260928 10:54:40.143024 20943 tablet_bootstrap.cc:654] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Neither blocks nor log segments found. Creating new log.
I20260928 10:54:40.143311 20941 tablet_bootstrap.cc:492] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: No bootstrap required, opened a new log
I20260928 10:54:40.143314 20942 ts_tablet_manager.cc:1397] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 10:54:40.143643 20941 ts_tablet_manager.cc:1397] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260928 10:54:40.144572 20943 log.cc:824] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:40.149282 20943 tablet_bootstrap.cc:492] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: No bootstrap required, opened a new log
I20260928 10:54:40.149763 20943 ts_tablet_manager.cc:1397] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260928 10:54:40.152755 20942 raft_consensus.cc:348] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.153298 20942 raft_consensus.cc:374] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:40.153431 20942 raft_consensus.cc:729] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef7295215fb4179818f49d684029947, State: Initialized, Role: FOLLOWER
I20260928 10:54:40.154220 20942 consensus_queue.cc:260] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.155175 20941 raft_consensus.cc:348] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.155687 20941 raft_consensus.cc:374] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:40.155819 20941 raft_consensus.cc:729] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f81d0b710d42a5b2f0f62aacb3f403, State: Initialized, Role: FOLLOWER
I20260928 10:54:40.156692 20942 ts_tablet_manager.cc:1428] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 10:54:40.156868 20941 consensus_queue.cc:260] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.158334 20792 heartbeater.cc:503] Master 127.5.227.190:44379 was elected leader, sending a full tablet report...
I20260928 10:54:40.159415 20941 ts_tablet_manager.cc:1428] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260928 10:54:40.162858 20943 raft_consensus.cc:348] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.163497 20943 raft_consensus.cc:374] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:54:40.163638 20943 raft_consensus.cc:729] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9273a6b555844ec9ca66403535ccb6a, State: Initialized, Role: FOLLOWER
I20260928 10:54:40.164544 20943 consensus_queue.cc:260] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.165836 20924 heartbeater.cc:503] Master 127.5.227.190:44379 was elected leader, sending a full tablet report...
I20260928 10:54:40.166641 20943 ts_tablet_manager.cc:1428] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
W20260928 10:54:40.182760 20661 tablet.cc:2369] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:54:40.192310 20947 raft_consensus.cc:482] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:54:40.192677 20947 raft_consensus.cc:504] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.194741 20947 leader_election.cc:288] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9273a6b555844ec9ca66403535ccb6a (127.5.227.131:40729), 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755)
I20260928 10:54:40.204756 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.195301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:36337 (local address 127.5.227.131:40729)
0928 10:54:40.195593 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.195602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.196850 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 10:54:40.197226 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.197232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.197307 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.197408 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:40.198263 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.199263 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.202394 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.203599 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.203873 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.203973 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.204323 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.204459 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 10:54:40.204895 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.195536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:40729 (local address 127.5.227.129:36337)
0928 10:54:40.196668 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.196701 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:40.197019 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:40.197436 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:40.197440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:40.197467 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:40.197932 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.197942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.199444 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.199452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.202156 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.202178 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.203916 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.203924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.204114 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.204164 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 10:54:40.204290 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 10:54:40.204447 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":76,"thread_start_us":545,"threads_started":1}
I20260928 10:54:40.205032 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.195445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:59551 (local address 127.5.227.130:43755)
0928 10:54:40.195665 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.195675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.197129 (+ 1454us) server_negotiation.cc:407] Connection header received
0928 10:54:40.197420 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.197424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.197488 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.197595 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:54:40.198418 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.199142 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.202651 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.203814 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.204046 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.204133 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.204525 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.204736 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:54:40.205300 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.195687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:43755 (local address 127.5.227.129:59551)
0928 10:54:40.196765 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.196782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:54:40.197249 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:54:40.197611 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:54:40.197616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:54:40.197664 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:54:40.198232 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.198245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.199358 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.199370 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.202181 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:54:40.202200 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.204091 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.204098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:54:40.204328 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.204363 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:54:40.204488 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 10:54:40.204668 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":128,"thread_start_us":589,"threads_started":1}
I20260928 10:54:40.206346 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee8c650b5a04f888164405cc15e21c8" candidate_uuid: "bef7295215fb4179818f49d684029947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9273a6b555844ec9ca66403535ccb6a" is_pre_election: true
I20260928 10:54:40.206770 20747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee8c650b5a04f888164405cc15e21c8" candidate_uuid: "bef7295215fb4179818f49d684029947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" is_pre_election: true
I20260928 10:54:40.206871 20879 raft_consensus.cc:2454] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef7295215fb4179818f49d684029947 in term 0.
I20260928 10:54:40.207242 20747 raft_consensus.cc:2454] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef7295215fb4179818f49d684029947 in term 0.
I20260928 10:54:40.208192 20548 leader_election.cc:302] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79f81d0b710d42a5b2f0f62aacb3f403, bef7295215fb4179818f49d684029947, c9273a6b555844ec9ca66403535ccb6a; no voters:
I20260928 10:54:40.208729 20947 raft_consensus.cc:2781] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:54:40.208906 20947 raft_consensus.cc:482] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:54:40.209036 20947 raft_consensus.cc:3037] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:40.211867 20947 raft_consensus.cc:504] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
I20260928 10:54:40.212909 20947 leader_election.cc:288] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [CANDIDATE]: Term 1 election: Requested vote from peers c9273a6b555844ec9ca66403535ccb6a (127.5.227.131:40729), 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755)
I20260928 10:54:40.213611 20747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee8c650b5a04f888164405cc15e21c8" candidate_uuid: "bef7295215fb4179818f49d684029947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f81d0b710d42a5b2f0f62aacb3f403"
I20260928 10:54:40.213855 20747 raft_consensus.cc:3037] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:40.213953 20879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee8c650b5a04f888164405cc15e21c8" candidate_uuid: "bef7295215fb4179818f49d684029947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9273a6b555844ec9ca66403535ccb6a"
I20260928 10:54:40.214445 20879 raft_consensus.cc:3037] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:54:40.217056 20747 raft_consensus.cc:2454] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef7295215fb4179818f49d684029947 in term 1.
I20260928 10:54:40.217502 20879 raft_consensus.cc:2454] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef7295215fb4179818f49d684029947 in term 1.
I20260928 10:54:40.218279 20548 leader_election.cc:302] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f81d0b710d42a5b2f0f62aacb3f403, bef7295215fb4179818f49d684029947; no voters:
I20260928 10:54:40.218967 20947 raft_consensus.cc:2781] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:54:40.219884 20947 raft_consensus.cc:686] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 1 LEADER]: Becoming Leader. State: Replica: bef7295215fb4179818f49d684029947, State: Running, Role: LEADER
I20260928 10:54:40.220664 20947 consensus_queue.cc:237] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } }
W20260928 10:54:40.223980 20793 tablet.cc:2369] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:54:40.226755 20473 catalog_manager.cc:5495] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 reported cstate change: term changed from 0 to 1, leader changed from <none> to bef7295215fb4179818f49d684029947 (127.5.227.129). New cstate: current_term: 1 leader_uuid: "bef7295215fb4179818f49d684029947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef7295215fb4179818f49d684029947" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 45513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 } health_report { overall_health: UNKNOWN } } }
W20260928 10:54:40.257782 20925 tablet.cc:2369] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:54:40.261405 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.253893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.5.227.129:45513)
0928 10:54:40.254133 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.254138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.254159 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:54:40.254345 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.254349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.254692 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.254865 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:54:40.255392 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.256509 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.258487 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.259041 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.259298 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.260010 (+ 712us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:54:40.260705 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:54:40.260835 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.261090 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.261175 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 10:54:40.272775 20747 raft_consensus.cc:1261] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Refusing update from remote peer bef7295215fb4179818f49d684029947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:40.273020 20879 raft_consensus.cc:1261] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 1 FOLLOWER]: Refusing update from remote peer bef7295215fb4179818f49d684029947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:54:40.274281 20952 consensus_queue.cc:1035] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f81d0b710d42a5b2f0f62aacb3f403" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 43755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:40.274781 20947 consensus_queue.cc:1035] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9273a6b555844ec9ca66403535ccb6a" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 40729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:54:40.291481 20956 mvcc.cc:205] Tried to move back new op lower bound from 7334268437584125952 to 7334268437388054528. Current Snapshot: MvccSnapshot[applied={T|T < 7334268437584125952}]
I20260928 10:54:40.310519 20474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:54:40.310940 20474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:54:40.315634 20474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Sending DeleteTablet for 3 replicas of tablet 9ee8c650b5a04f888164405cc15e21c8
I20260928 10:54:40.317176 20859 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee8c650b5a04f888164405cc15e21c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:40 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:54760
I20260928 10:54:40.317258 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee8c650b5a04f888164405cc15e21c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:40 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:56080
I20260928 10:54:40.317258 20727 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee8c650b5a04f888164405cc15e21c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:54:40 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:57014
I20260928 10:54:40.317770 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 20511
I20260928 10:54:40.323485 20967 tablet_replica.cc:330] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: stopping tablet replica
I20260928 10:54:40.328152 20966 tablet_replica.cc:330] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: stopping tablet replica
I20260928 10:54:40.328866 20966 raft_consensus.cc:2229] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:54:40.329833 20966 raft_consensus.cc:2258] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:40.330683 20967 raft_consensus.cc:2229] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:54:40.331791 20967 raft_consensus.cc:2258] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:40.332631 20965 tablet_replica.cc:330] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: stopping tablet replica
I20260928 10:54:40.333408 20965 raft_consensus.cc:2229] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:54:40.333982 20965 raft_consensus.cc:2258] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:54:40.336613 20965 ts_tablet_manager.cc:1905] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:40.336809 20966 ts_tablet_manager.cc:1905] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:40.334430 20967 ts_tablet_manager.cc:1905] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:54:40.343847 20966 ts_tablet_manager.cc:1918] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:40.344065 20966 log.cc:1196] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/9ee8c650b5a04f888164405cc15e21c8
I20260928 10:54:40.344290 20965 ts_tablet_manager.cc:1918] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:40.344456 20965 log.cc:1196] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/9ee8c650b5a04f888164405cc15e21c8
I20260928 10:54:40.344736 20966 ts_tablet_manager.cc:1939] T 9ee8c650b5a04f888164405cc15e21c8 P 79f81d0b710d42a5b2f0f62aacb3f403: Deleting consensus metadata
I20260928 10:54:40.345050 20965 ts_tablet_manager.cc:1939] T 9ee8c650b5a04f888164405cc15e21c8 P c9273a6b555844ec9ca66403535ccb6a: Deleting consensus metadata
I20260928 10:54:40.346997 20458 catalog_manager.cc:4841] TS 79f81d0b710d42a5b2f0f62aacb3f403 (127.5.227.130:43755): tablet 9ee8c650b5a04f888164405cc15e21c8 (table test-table [id=b385e002552e42028b77f4fafa713bdb]) successfully deleted
I20260928 10:54:40.347025 20459 catalog_manager.cc:4841] TS c9273a6b555844ec9ca66403535ccb6a (127.5.227.131:40729): tablet 9ee8c650b5a04f888164405cc15e21c8 (table test-table [id=b385e002552e42028b77f4fafa713bdb]) successfully deleted
I20260928 10:54:40.351025 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.318984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.5.227.129:45513)
0928 10:54:40.319254 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.319265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.319478 (+ 213us) server_negotiation.cc:407] Connection header received
0928 10:54:40.319808 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.319818 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.319942 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.320101 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:40.322684 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.323965 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.332561 (+ 8596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.333646 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.333865 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.338911 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:40.338935 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:40.338955 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:40.339008 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:40.346582 (+ 7574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.348532 (+ 1950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.348543 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.348546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.348784 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.349234 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.349239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.349243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.349738 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:40.350077 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.350408 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.350677 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:54:40.351902 20655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:40.352614 20967 ts_tablet_manager.cc:1918] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:54:40.352886 20967 log.cc:1196] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/9ee8c650b5a04f888164405cc15e21c8
I20260928 10:54:40.455624 20967 ts_tablet_manager.cc:1939] T 9ee8c650b5a04f888164405cc15e21c8 P bef7295215fb4179818f49d684029947: Deleting consensus metadata
I20260928 10:54:40.455619 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 20511
I20260928 10:54:40.481222 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 20664
W20260928 10:54:40.481496 20458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.227.129:45513 (error 108)
I20260928 10:54:40.484174 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.483676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:54948)
0928 10:54:40.483894 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.484016 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 10:54:40.484520 20458 catalog_manager.cc:4584] TS bef7295215fb4179818f49d684029947 (127.5.227.129:45513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ee8c650b5a04f888164405cc15e21c8: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
I20260928 10:54:40.500038 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.482544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.5.227.130:43755)
0928 10:54:40.482752 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.482761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.482784 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:54:40.482858 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.482867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.482961 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.483126 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:40.485120 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.486247 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.488528 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.489084 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.489213 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.492792 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:40.492807 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:40.492825 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:40.492867 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:40.496323 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.497325 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.497334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.497336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.497507 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.498096 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.498100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.498103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.498619 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:40.498870 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.499287 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.499506 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:54:40.500855 20787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:40.538344 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.537761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:54958)
0928 10:54:40.538017 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.538165 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 10:54:40.539688 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.539266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:54972)
0928 10:54:40.539445 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.539557 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 10:54:40.601251 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 20664
I20260928 10:54:40.601560 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.601015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:54982)
0928 10:54:40.601223 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.601320 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 10:54:40.602954 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.602431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:54990)
0928 10:54:40.602666 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.602776 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 10:54:40.622552 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 20796
I20260928 10:54:40.638700 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.623441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.5.227.131:40729)
0928 10:54:40.623697 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:40.623704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:40.623769 (+ 65us) server_negotiation.cc:407] Connection header received
0928 10:54:40.623843 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:40.623846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:40.623928 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:40.624050 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:40.625461 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.626231 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.627605 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:40.628130 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:40.628274 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:40.631417 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:40.631435 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:40.631453 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:40.631500 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:40.635344 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.636341 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.636349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.636352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.636523 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:40.636950 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:40.636956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:40.636959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:40.637351 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:40.637691 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:40.638031 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 10:54:40.638282 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 10:54:40.639871 20919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:40.684845 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.684294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:55006)
0928 10:54:40.684550 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.684671 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 10:54:40.686295 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:40.685791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:45513 (local address 127.0.0.1:55012)
0928 10:54:40.686061 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:40.686151 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 10:54:40.740511 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 20796
I20260928 10:54:40.761417 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 20441
I20260928 10:54:40.762833 20504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:40.887420 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 20441
I20260928 10:54:40.912221 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44379
--webserver_interface=127.5.227.190
--webserver_port=36719
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:41.286679 20973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:41.287012 20973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:41.287133 20973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:41.296697 20973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:54:41.296849 20973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:41.296916 20973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:41.297086 20973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:54:41.297188 20973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:54:41.308238 20973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:41.309656 20973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:41.310973 20973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:41.324440 20978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:41.324549 20979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:41.324544 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:41.326172 20973 server_base.cc:1034] running on GCE node
I20260928 10:54:41.327510 20973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:41.329430 20973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:41.330744 20973 hybrid_clock.cc:648] HybridClock initialized: now 1790592881330692 us; error 48 us; skew 500 ppm
I20260928 10:54:41.331352 20973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:41 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592881, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:41.335973 20973 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:54:41.337882 20973 webserver.cc:460] Webserver started at http://127.5.227.190:36719/ using document root <none> and password file <none>
I20260928 10:54:41.338672 20973 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:41.338842 20973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:41.345434 20973 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:54:41.352017 20988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:41.354043 20973 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:54:41.354334 20973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "39346fd2b3bc46b2a5fc38bfebc979de"
format_stamp: "Formatted at 2026-09-28 10:54:34 on dist-test-slave-8w8g"
server_key: "f54af56296eb2a8a71d240a12704a731"
server_key_iv: "45f9877e5bf35b495d3aeb6b9bbfe854"
server_key_version: "encryptionkey@0"
I20260928 10:54:41.355995 20973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:41.373881 20973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:41.379518 20973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:41.379892 20973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:41.403252 20973 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:44379
I20260928 10:54:41.403276 21039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:44379 every 8 connection(s)
I20260928 10:54:41.405063 20973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:54:41.410837 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 20973
I20260928 10:54:41.412070 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:45513
--local_ip_for_outbound_sockets=127.5.227.129
--tserver_master_addrs=127.5.227.190:44379
--webserver_port=33421
--webserver_interface=127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:41.415529 21040 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:54:41.421736 21040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Bootstrap starting.
I20260928 10:54:41.445706 21040 log.cc:824] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Log is configured to *not* fsync() on all Append() calls
I20260928 10:54:41.465689 21040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:54:41.466311 21040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de: Bootstrap complete.
I20260928 10:54:41.475970 21040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:41.476950 21040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39346fd2b3bc46b2a5fc38bfebc979de, State: Initialized, Role: FOLLOWER
I20260928 10:54:41.477685 21040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:41.477953 21040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:54:41.478077 21040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:54:41.478286 21040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 2 FOLLOWER]: Advancing to term 3
I20260928 10:54:41.482604 21040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:41.483213 21040 leader_election.cc:302] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39346fd2b3bc46b2a5fc38bfebc979de; no voters:
I20260928 10:54:41.484053 21040 leader_election.cc:288] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 10:54:41.484220 21044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 3 FOLLOWER]: Leader election won for term 3
I20260928 10:54:41.485433 21044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [term 3 LEADER]: Becoming Leader. State: Replica: 39346fd2b3bc46b2a5fc38bfebc979de, State: Running, Role: LEADER
I20260928 10:54:41.486182 21044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } }
I20260928 10:54:41.486673 21040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:54:41.489331 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39346fd2b3bc46b2a5fc38bfebc979de. Latest consensus state: current_term: 3 leader_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } } }
I20260928 10:54:41.490091 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:41.491058 21045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39346fd2b3bc46b2a5fc38bfebc979de" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44379 } } }
I20260928 10:54:41.491396 21045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39346fd2b3bc46b2a5fc38bfebc979de [sys.catalog]: This master's current role is: LEADER
I20260928 10:54:41.501041 21051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:54:41.511950 21051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:54:41.514037 21051 catalog_manager.cc:1259] Loaded cluster ID: 2820cc864c0f424aa06e8f2dfd9ffac7
I20260928 10:54:41.514181 21051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 10:54:41.516520 21051 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb509cea9ad google::LogMessage::Fail() at ??:0
@ 0x7fb509ced88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb509cea4de google::LogMessage::Flush() at ??:0
@ 0x7fb509ceba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb51f71d512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb51f71a976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb51f7a2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb51f7a2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb50c936345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb50c93991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb50c939799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb50c914eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb50fbee6db start_thread at ??:0
@ 0x7fb508b8461f clone at ??:0
W20260928 10:54:41.854207 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:41.854612 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:41.854827 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:41.865087 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:41.865300 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:41.865501 21042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:54:41.877389 21042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:45513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=33421
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:41.879114 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:41.880694 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:41.898435 21085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:41.898705 21082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:41.898727 21083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:41.899080 21042 server_base.cc:1034] running on GCE node
I20260928 10:54:41.900131 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:41.901319 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:41.902691 21042 hybrid_clock.cc:648] HybridClock initialized: now 1790592881902622 us; error 53 us; skew 500 ppm
I20260928 10:54:41.903133 21042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:41 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592881, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:41.907163 21042 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:54:41.909524 21042 webserver.cc:460] Webserver started at http://127.5.227.129:33421/ using document root <none> and password file <none>
I20260928 10:54:41.910373 21042 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:41.910540 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:41.917387 21042 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:54:41.921188 21092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:41.923600 21042 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:54:41.923967 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "bef7295215fb4179818f49d684029947"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "fb7344a41595620147dde7e0ee4e4cc7"
server_key_iv: "8d2196114f3b8f49fa3f3a782ac82302"
server_key_version: "encryptionkey@0"
I20260928 10:54:41.925130 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:41.946789 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:41.952211 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:41.952625 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:41.954030 21042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:41.955806 21042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:41.955953 21042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:41.956121 21042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:41.956221 21042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:41.995915 21042 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:45513
I20260928 10:54:41.995936 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:45513 every 8 connection(s)
I20260928 10:54:41.997519 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:54:42.003319 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:42.001047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:47623)
0928 10:54:42.001971 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:42.002348 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":431,"threads_started":1}
W20260928 10:54:42.004319 21205 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:44379 (0 consecutive failures): Network error: Failed to ping master at 127.5.227.190:44379: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
I20260928 10:54:42.005748 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 21042
I20260928 10:54:42.005987 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:42.005268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:41423)
0928 10:54:42.005513 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:42.005795 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 10:54:42.006680 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:43755
--local_ip_for_outbound_sockets=127.5.227.130
--tserver_master_addrs=127.5.227.190:44379
--webserver_port=38809
--webserver_interface=127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:54:42.007261 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:42.006690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:44609)
0928 10:54:42.006996 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:42.007129 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260928 10:54:42.007714 21205 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:54:42.381647 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:42.381982 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:42.382097 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:42.390189 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:42.390339 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:42.390462 21208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:54:42.400636 21208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:43755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=38809
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:42.401943 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:42.403370 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:42.418710 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:42.418735 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:42.419636 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:42.420325 21208 server_base.cc:1034] running on GCE node
I20260928 10:54:42.421130 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:42.422325 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:42.423699 21208 hybrid_clock.cc:648] HybridClock initialized: now 1790592882423602 us; error 84 us; skew 500 ppm
I20260928 10:54:42.424256 21208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:42 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592882, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:42.427968 21208 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:54:42.429651 21208 webserver.cc:460] Webserver started at http://127.5.227.130:38809/ using document root <none> and password file <none>
I20260928 10:54:42.430425 21208 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:42.430532 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:42.437430 21208 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:54:42.440935 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:42.442850 21208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:54:42.443097 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "79f81d0b710d42a5b2f0f62aacb3f403"
format_stamp: "Formatted at 2026-09-28 10:54:35 on dist-test-slave-8w8g"
server_key: "9ed12373ca3a72a8b3c95c70a54cacb1"
server_key_iv: "e988b69ad5071eb3b75bb268f59bbf86"
server_key_version: "encryptionkey@0"
I20260928 10:54:42.444531 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:42.463670 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:42.469689 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:42.469980 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:42.471177 21208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:42.473109 21208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:42.473237 21208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:42.473322 21208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:42.473387 21208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:42.514328 21208 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:43755
I20260928 10:54:42.514541 21335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:43755 every 8 connection(s)
I20260928 10:54:42.516155 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:54:42.522228 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:42.520044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:42693)
0928 10:54:42.520925 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:42.521306 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":385,"threads_started":1}
W20260928 10:54:42.523249 21336 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:44379 (0 consecutive failures): Network error: Failed to ping master at 127.5.227.190:44379: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
I20260928 10:54:42.524379 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:42.523971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:49867)
0928 10:54:42.524186 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:42.524267 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 10:54:42.525506 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:42.525095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:35171)
0928 10:54:42.525268 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:42.525363 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 10:54:42.525835 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 21208
W20260928 10:54:42.525866 21336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 10:54:42.526486 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:40729
--local_ip_for_outbound_sockets=127.5.227.131
--tserver_master_addrs=127.5.227.190:44379
--webserver_port=44121
--webserver_interface=127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:54:42.940642 21339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:54:42.940972 21339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:54:42.941087 21339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:54:42.950762 21339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:54:42.950915 21339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:54:42.951043 21339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:54:42.965276 21339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:40729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=44121
--tserver_master_addrs=127.5.227.190:44379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:54:42.966900 21339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:54:42.968472 21339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:54:42.986354 21347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:42.986465 21345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:54:42.986523 21344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:54:42.987327 21339 server_base.cc:1034] running on GCE node
I20260928 10:54:42.988160 21339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:54:42.989686 21339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:54:42.991068 21339 hybrid_clock.cc:648] HybridClock initialized: now 1790592882990992 us; error 90 us; skew 500 ppm
I20260928 10:54:42.991535 21339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:54:42 dist-test-slave-8w8g krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592882, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:54:42.996124 21339 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:54:42.998108 21339 webserver.cc:460] Webserver started at http://127.5.227.131:44121/ using document root <none> and password file <none>
I20260928 10:54:42.998919 21339 fs_manager.cc:359] Metadata directory not provided
I20260928 10:54:42.999091 21339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:54:43.006122 21339 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:54:43.009716 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:43.008570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:39357)
0928 10:54:43.009195 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:43.009361 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":313,"threads_started":1}
I20260928 10:54:43.009876 21355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:43.011651 21339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 10:54:43.011920 21339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "c9273a6b555844ec9ca66403535ccb6a"
format_stamp: "Formatted at 2026-09-28 10:54:36 on dist-test-slave-8w8g"
server_key: "d4a1af75c95fd7471e91fab675973489"
server_key_iv: "1c809de53d135da1be2d39f0b94f9c39"
server_key_version: "encryptionkey@0"
I20260928 10:54:43.013016 21339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:54:43.032491 21339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:54:43.038892 21339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:54:43.039364 21339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:54:43.040868 21339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:54:43.042438 21339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:54:43.042557 21339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:43.042639 21339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:54:43.042687 21339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:54:43.087219 21339 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:40729
I20260928 10:54:43.087350 21467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:40729 every 8 connection(s)
I20260928 10:54:43.088908 21339 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:54:43.094731 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:43.092631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:34713)
0928 10:54:43.093509 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:43.093761 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":369,"threads_started":1}
W20260928 10:54:43.095986 21468 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:44379 (0 consecutive failures): Network error: Failed to ping master at 127.5.227.190:44379: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
I20260928 10:54:43.096379 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 21339
I20260928 10:54:43.097533 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:43.096962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:36541)
0928 10:54:43.097305 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:43.097394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260928 10:54:43.098728 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:43.098275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:47995)
0928 10:54:43.098482 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:43.098575 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260928 10:54:43.099186 21468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 10:54:43.527684 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:43.526729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:57453)
0928 10:54:43.527278 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:43.527398 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":266,"threads_started":1}
I20260928 10:54:44.011958 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:44.011053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:47041)
0928 10:54:44.011592 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:44.011695 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":256,"threads_started":1}
I20260928 10:54:44.100855 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:44.099973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:45459)
0928 10:54:44.100511 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:44.100623 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":269,"threads_started":1}
I20260928 10:54:44.530094 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:44.529150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:53683)
0928 10:54:44.529735 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:44.529828 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":320,"threads_started":1}
I20260928 10:54:45.014081 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:45.013242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:44593)
0928 10:54:45.013805 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:45.013898 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":327,"threads_started":1}
I20260928 10:54:45.103372 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:45.102167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:51675)
0928 10:54:45.102923 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:45.103164 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":315,"threads_started":1}
I20260928 10:54:45.532402 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:45.531225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:52081)
0928 10:54:45.531918 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:45.532162 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":382,"threads_started":1}
I20260928 10:54:46.016701 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:46.015468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:52693)
0928 10:54:46.016373 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:46.016478 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":638,"threads_started":1}
I20260928 10:54:46.105667 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:46.104824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:46111)
0928 10:54:46.105383 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:46.105484 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":323,"threads_started":1}
I20260928 10:54:46.534614 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:46.533651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:54241)
0928 10:54:46.534257 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:46.534363 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":322,"threads_started":1}
I20260928 10:54:47.019218 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:47.018289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:34915)
0928 10:54:47.018899 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:47.018999 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":284,"threads_started":1}
I20260928 10:54:47.107895 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:47.106935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:59261)
0928 10:54:47.107518 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:47.107622 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":324,"threads_started":1}
I20260928 10:54:47.536958 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:47.535991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:53771)
0928 10:54:47.536656 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:47.536753 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260928 10:54:48.021425 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:48.020392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:48497)
0928 10:54:48.020951 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:48.021176 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":333,"threads_started":1}
I20260928 10:54:48.110316 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:48.109302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:40857)
0928 10:54:48.109916 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:48.110099 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":338,"threads_started":1}
I20260928 10:54:48.538923 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:48.538004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:38209)
0928 10:54:48.538630 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:48.538741 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":302,"threads_started":1}
I20260928 10:54:49.023531 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:49.022569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:36415)
0928 10:54:49.023156 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:49.023260 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":249,"threads_started":1}
I20260928 10:54:49.112510 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:49.111601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:37227)
0928 10:54:49.112215 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:49.112318 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":364,"threads_started":1}
I20260928 10:54:49.541292 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:49.540423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:33223)
0928 10:54:49.540981 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:49.541072 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":254,"threads_started":1}
I20260928 10:54:50.025672 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:50.024754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:53639)
0928 10:54:50.025380 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:50.025479 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":403,"threads_started":1}
I20260928 10:54:50.114883 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:50.113864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:54119)
0928 10:54:50.114509 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:50.114595 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":328,"threads_started":1}
I20260928 10:54:50.543514 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:50.542590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:42905)
0928 10:54:50.543233 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:50.543340 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":322,"threads_started":1}
I20260928 10:54:51.027840 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:51.026838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:44775)
0928 10:54:51.027468 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:51.027555 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":367,"threads_started":1}
I20260928 10:54:51.117239 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:51.116260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:57331)
0928 10:54:51.116900 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:51.117005 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":264,"threads_started":1}
I20260928 10:54:51.545609 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:51.544762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:53663)
0928 10:54:51.545321 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:51.545425 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":303,"threads_started":1}
I20260928 10:54:52.030435 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:52.029482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:53041)
0928 10:54:52.030068 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:52.030180 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":301,"threads_started":1}
I20260928 10:54:52.119215 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:52.118444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:38007)
0928 10:54:52.118942 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:52.119042 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":238,"threads_started":1}
I20260928 10:54:52.547977 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:52.547059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:38369)
0928 10:54:52.547612 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:52.547706 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
I20260928 10:54:53.032368 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:53.031501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:53251)
0928 10:54:53.032075 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:53.032177 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":310,"threads_started":1}
I20260928 10:54:53.121378 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:53.120542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:38415)
0928 10:54:53.121106 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:53.121201 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
I20260928 10:54:53.550192 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:53.549248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:43033)
0928 10:54:53.549811 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:53.549962 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":325,"threads_started":1}
I20260928 10:54:54.034279 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:54.033471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:33359)
0928 10:54:54.034016 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:54.034107 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":258,"threads_started":1}
I20260928 10:54:54.123495 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:54.122620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:52921)
0928 10:54:54.123183 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:54.123273 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":283,"threads_started":1}
I20260928 10:54:54.552192 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:54.551350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:50429)
0928 10:54:54.551922 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:54.552020 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":282,"threads_started":1}
I20260928 10:54:55.036458 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:55.035488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:59419)
0928 10:54:55.036138 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:55.036252 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":356,"threads_started":1}
I20260928 10:54:55.125504 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:55.124567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:48465)
0928 10:54:55.125240 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:55.125340 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":290,"threads_started":1}
I20260928 10:54:55.554051 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:55.553195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:37737)
0928 10:54:55.553740 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:55.553840 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":287,"threads_started":1}
I20260928 10:54:56.038455 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:56.037653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:35833)
0928 10:54:56.038166 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:56.038263 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260928 10:54:56.127816 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:56.126758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:32995)
0928 10:54:56.127363 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:56.127558 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":301,"threads_started":1}
I20260928 10:54:56.556474 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:56.555260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:55197)
0928 10:54:56.555955 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:56.556053 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":336,"threads_started":1}
I20260928 10:54:57.041003 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:57.040082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:39051)
0928 10:54:57.040688 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:57.040790 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":334,"threads_started":1}
I20260928 10:54:57.129897 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:57.128965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:35793)
0928 10:54:57.129623 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:57.129711 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":416,"threads_started":1}
I20260928 10:54:57.558833 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:57.557894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.130:42985)
0928 10:54:57.558524 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:57.558628 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":318,"threads_started":1}
I20260928 10:54:58.043309 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:58.042260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.129:36379)
0928 10:54:58.042887 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:58.043139 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":358,"threads_started":1}
I20260928 10:54:58.108639 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 21042
I20260928 10:54:58.132171 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:58.131282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:44379 (local address 127.5.227.131:43351)
0928 10:54:58.131879 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 10:54:58.131975 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":305,"threads_started":1}
I20260928 10:54:58.142143 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:58.110225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.5.227.129:45513)
0928 10:54:58.111210 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:58.111260 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:58.111326 (+ 66us) server_negotiation.cc:407] Connection header received
0928 10:54:58.113913 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:58.113927 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:58.114443 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:58.114878 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:58.116827 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:58.118918 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:58.121067 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:58.121948 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:58.122321 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:58.133138 (+ 10817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:58.133149 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:58.133169 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:58.133226 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:58.137221 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:58.138233 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:58.138241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:58.138249 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:58.138440 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:58.139082 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:58.139088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:58.139090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:58.139887 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:58.140538 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:58.141309 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 10:54:58.141804 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":437,"threads_started":1}
I20260928 10:54:58.144009 21200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:58.256245 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 21042
I20260928 10:54:58.278874 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 21208
I20260928 10:54:58.303676 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:58.280225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.5.227.130:43755)
0928 10:54:58.280799 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:58.280814 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:58.280853 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:54:58.282960 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:58.282973 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:58.283483 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:58.283962 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:58.286266 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:58.288214 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:58.289746 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:58.290760 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:58.291156 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:58.294770 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:58.294780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:58.294800 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:58.294855 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:58.298939 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:58.300217 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:58.300226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:58.300234 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:58.300472 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:58.300973 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:58.300977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:58.300980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:58.301651 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:58.301904 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:58.302650 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 10:54:58.303365 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":264,"threads_started":1}
I20260928 10:54:58.305586 21331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:58.409230 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 21208
I20260928 10:54:58.431316 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 21339
I20260928 10:54:58.452733 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:54:58.432587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.5.227.131:40729)
0928 10:54:58.433380 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 10:54:58.433394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:54:58.433473 (+ 79us) server_negotiation.cc:407] Connection header received
0928 10:54:58.435120 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:54:58.435130 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:54:58.435566 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:54:58.435964 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:54:58.437518 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:58.439287 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:58.441109 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:54:58.441837 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:54:58.442024 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:54:58.444962 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:54:58.444971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:54:58.444993 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:54:58.445037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:54:58.448387 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:58.449303 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:58.449311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:58.449319 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:58.449532 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:54:58.450107 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:54:58.450112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:54:58.450114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:54:58.450790 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:54:58.451076 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 10:54:58.451690 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 10:54:58.452278 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":504,"threads_started":1}
I20260928 10:54:58.454506 21463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:54:58.548852 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 21339
2026-09-28T10:54:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25985 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:54:58 dist-test-slave-8w8g krb5kdc[21525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:54:58 dist-test-slave-8w8g krb5kdc[21525](info): set up 2 sockets
Sep 28 10:54:58 dist-test-slave-8w8g krb5kdc[21525](info): commencing operation
krb5kdc: starting...
W20260928 10:54:59.802675 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:55:00 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:55:00.134776 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36149
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:55:00.564803 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:00.565161 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:00.565258 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:00.573505 21538 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 10:55:00.573657 21538 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 10:55:00.573697 21538 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 10:55:00.573742 21538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:55:00.573881 21538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:00.574000 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:00.574091 21538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:55:00.574165 21538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:55:00.588267 21538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:00.589722 21538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:00.591078 21538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:00.607077 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:00.607153 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:00.607118 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:00.608170 21538 server_base.cc:1034] running on GCE node
I20260928 10:55:00.609161 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:00.610776 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:00.612036 21538 hybrid_clock.cc:648] HybridClock initialized: now 1790592900611936 us; error 59 us; skew 500 ppm
I20260928 10:55:00.612385 21538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:00 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592900, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:00.616294 21538 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:55:00.618465 21538 webserver.cc:460] Webserver started at http://127.5.227.190:39215/ using document root <none> and password file <none>
I20260928 10:55:00.619266 21538 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:00.619383 21538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:00.619670 21538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:55:00.623106 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "00059920d8414debbe150b9172f45d9d"
format_stamp: "Formatted at 2026-09-28 10:55:00 on dist-test-slave-8w8g"
server_key: "b16f8fd097dcd7ab90c1117dd2ee0965"
server_key_iv: "2d6a1479e2c2043dd63c6194caab8607"
server_key_version: "encryptionkey@0"
I20260928 10:55:00.624091 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "00059920d8414debbe150b9172f45d9d"
format_stamp: "Formatted at 2026-09-28 10:55:00 on dist-test-slave-8w8g"
server_key: "b16f8fd097dcd7ab90c1117dd2ee0965"
server_key_iv: "2d6a1479e2c2043dd63c6194caab8607"
server_key_version: "encryptionkey@0"
I20260928 10:55:00.630224 21538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:55:00.634912 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:00.637048 21538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:55:00.637279 21538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "00059920d8414debbe150b9172f45d9d"
format_stamp: "Formatted at 2026-09-28 10:55:00 on dist-test-slave-8w8g"
server_key: "b16f8fd097dcd7ab90c1117dd2ee0965"
server_key_iv: "2d6a1479e2c2043dd63c6194caab8607"
server_key_version: "encryptionkey@0"
I20260928 10:55:00.637557 21538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:00.651165 21538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:00.657027 21538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:00.657351 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:00.680208 21604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:36149 every 8 connection(s)
I20260928 10:55:00.680415 21538 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:36149
I20260928 10:55:00.682394 21538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:55:00.687290 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:55:00.689774 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 21538
I20260928 10:55:00.690099 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:55:00.691462 6030 external_mini_cluster.cc:1428] Setting key 9b45a5fabdf6fd81baeb3b57f8c4234f
I20260928 10:55:00.703297 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Bootstrap starting.
I20260928 10:55:00.707968 21605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Neither blocks nor log segments found. Creating new log.
Sep 28 10:55:00 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:55:00.709383 21605 log.cc:824] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:00.713716 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: No bootstrap required, opened a new log
I20260928 10:55:00.722270 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:00.693833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.5.227.190:36149)
0928 10:55:00.694788 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:00.694802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:00.694853 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:55:00.696758 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:00.696785 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:00.697248 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:00.697605 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:00.700109 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:00.701655 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:00.703919 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:00.704698 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:00.704873 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:00.710652 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:00.710665 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:00.710692 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:00.710759 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:00.715417 (+ 4658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:00.716626 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:00.716635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:00.716644 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:00.717180 (+ 536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:00.718003 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:00.718007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:00.718009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:00.718736 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:00.718987 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:00.719958 (+ 971us) server_negotiation.cc:299] Negotiation successful
0928 10:55:00.720420 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":484,"threads_started":1}
I20260928 10:55:00.727448 21605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:00.727937 21605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:55:00.728119 21605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00059920d8414debbe150b9172f45d9d, State: Initialized, Role: FOLLOWER
I20260928 10:55:00.728871 21605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:00.729169 21605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:55:00.729348 21605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:55:00.729583 21605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:55:00.732060 21605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:00.732640 21605 leader_election.cc:302] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00059920d8414debbe150b9172f45d9d; no voters:
I20260928 10:55:00.733320 21605 leader_election.cc:288] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:55:00.733522 21610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:55:00.734756 21610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 LEADER]: Becoming Leader. State: Replica: 00059920d8414debbe150b9172f45d9d, State: Running, Role: LEADER
I20260928 10:55:00.735371 21610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:00.736797 21605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:55:00.740145 21611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00059920d8414debbe150b9172f45d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } } }
I20260928 10:55:00.740698 21611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: This master's current role is: LEADER
I20260928 10:55:00.744051 21612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00059920d8414debbe150b9172f45d9d. Latest consensus state: current_term: 1 leader_uuid: "00059920d8414debbe150b9172f45d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } } }
I20260928 10:55:00.744393 21612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: This master's current role is: LEADER
I20260928 10:55:00.745069 21618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:55:00.750399 21618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:55:00.762090 21618 catalog_manager.cc:1347] Generated new cluster ID: ee071bd6874f4f8ea2a9bb16548519d8
I20260928 10:55:00.762244 21618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:55:00.804716 21618 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:55:00.806510 21618 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:55:00.821609 21618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Generated new TSK 0
I20260928 10:55:00.822646 21618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:55:00.909128 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36149
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:55:01.287163 21631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:01.287497 21631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:01.287674 21631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:01.296518 21631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:01.296675 21631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:01.296859 21631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:55:01.307638 21631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:01.309625 21631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:01.311177 21631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:01.327083 21639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:01.327042 21637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:01.328341 21636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:01.329607 21631 server_base.cc:1034] running on GCE node
I20260928 10:55:01.330318 21631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:01.331401 21631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:01.332697 21631 hybrid_clock.cc:648] HybridClock initialized: now 1790592901332610 us; error 75 us; skew 500 ppm
I20260928 10:55:01.333091 21631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:01 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:01.337041 21631 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:55:01.339011 21631 webserver.cc:460] Webserver started at http://127.5.227.129:46025/ using document root <none> and password file <none>
I20260928 10:55:01.339866 21631 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:01.340150 21631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:01.340569 21631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:55:01.344378 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "5f74db17a0dd4281b8262486ff8b08c4"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "96c84353c636c1fe65c8207ced1e7525"
server_key_iv: "d7d99ac6732d2c5c8c4f6ba12f26b677"
server_key_version: "encryptionkey@0"
I20260928 10:55:01.345431 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "5f74db17a0dd4281b8262486ff8b08c4"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "96c84353c636c1fe65c8207ced1e7525"
server_key_iv: "d7d99ac6732d2c5c8c4f6ba12f26b677"
server_key_version: "encryptionkey@0"
I20260928 10:55:01.352094 21631 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 10:55:01.356653 21646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:01.358670 21631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:55:01.359004 21631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "5f74db17a0dd4281b8262486ff8b08c4"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "96c84353c636c1fe65c8207ced1e7525"
server_key_iv: "d7d99ac6732d2c5c8c4f6ba12f26b677"
server_key_version: "encryptionkey@0"
I20260928 10:55:01.359272 21631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:01.378834 21631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:01.384404 21631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:01.384752 21631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:01.385965 21631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:01.387657 21631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:01.387856 21631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 10:55:01.388033 21631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:01.388129 21631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:01.430828 21631 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:32825
I20260928 10:55:01.430886 21758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:32825 every 8 connection(s)
I20260928 10:55:01.432379 21631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:55:01.440295 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 21631
I20260928 10:55:01.440575 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:55:01.441339 6030 external_mini_cluster.cc:1428] Setting key bce26979ec1cebd44fe20a56c7345f0f
Sep 28 10:55:01 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:55:01.461257 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:01.435798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:40799 (local address 127.5.227.190:36149)
0928 10:55:01.436332 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:01.436337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:01.437185 (+ 848us) server_negotiation.cc:407] Connection header received
0928 10:55:01.439241 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:01.439248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:01.439351 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:01.439485 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:01.441725 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:01.442899 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:01.445647 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:01.446181 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:01.446299 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:01.451556 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:01.451566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:01.451574 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:01.451628 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:01.456251 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:01.457758 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:01.457764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:01.457767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:01.457932 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:01.458715 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:01.458720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:01.458722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:01.459050 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:01.459188 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:01.460634 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0928 10:55:01.460859 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":285,"threads_started":1}
I20260928 10:55:01.462831 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:01.436161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:40799)
0928 10:55:01.436960 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:01.437001 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:01.438927 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:01.439821 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:01.439835 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:01.440551 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:55:01.441528 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:01.441552 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:01.443160 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:01.443165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:01.445433 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:01.445459 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:01.446472 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:01.446482 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:01.446904 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:01.447850 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:55:01.447887 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:55:01.451019 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:55:01.456485 (+ 5466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:01.456501 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:01.456525 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:01.457544 (+ 1019us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:01.458180 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:01.458188 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:01.458196 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:01.458538 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:01.459208 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:55:01.459221 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:55:01.459819 (+ 598us) client_negotiation.cc:769] Sending connection context
0928 10:55:01.460214 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 10:55:01.460924 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":308,"threads_started":1}
I20260928 10:55:01.466497 21759 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36149
I20260928 10:55:01.467242 21759 heartbeater.cc:464] Registering TS with master...
I20260928 10:55:01.468271 21759 heartbeater.cc:511] Master 127.5.227.190:36149 requested a full tablet report, sending...
I20260928 10:55:01.470919 21570 ts_manager.cc:194] Registered new tserver with Master: 5f74db17a0dd4281b8262486ff8b08c4 (127.5.227.129:32825)
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:55:01.472726 21570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:40799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:55:01.514285 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36149
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:55:01.907953 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:01.908337 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:01.908517 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:01.916162 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:01.916352 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:01.916551 21765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:55:01.927130 21765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:01.928601 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:01.930167 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:01.945294 21770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:01.945154 21771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:01.945459 21773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:01.946310 21765 server_base.cc:1034] running on GCE node
I20260928 10:55:01.947016 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:01.948173 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:01.949412 21765 hybrid_clock.cc:648] HybridClock initialized: now 1790592901949362 us; error 37 us; skew 500 ppm
I20260928 10:55:01.949818 21765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:01 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:01.953502 21765 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:55:01.955332 21765 webserver.cc:460] Webserver started at http://127.5.227.130:45419/ using document root <none> and password file <none>
I20260928 10:55:01.956166 21765 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:01.956307 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:01.956660 21765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:55:01.960139 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "06ad30337d914ff0a00965e2704fa6df"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "1c3f69b259662ce83e0894723e6ab75a"
server_key_iv: "cbadd871e5151ba4b26f72496120dacf"
server_key_version: "encryptionkey@0"
I20260928 10:55:01.961104 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "06ad30337d914ff0a00965e2704fa6df"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "1c3f69b259662ce83e0894723e6ab75a"
server_key_iv: "cbadd871e5151ba4b26f72496120dacf"
server_key_version: "encryptionkey@0"
I20260928 10:55:01.967191 21765 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:55:01.971819 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:01.973868 21765 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:55:01.974468 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "06ad30337d914ff0a00965e2704fa6df"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "1c3f69b259662ce83e0894723e6ab75a"
server_key_iv: "cbadd871e5151ba4b26f72496120dacf"
server_key_version: "encryptionkey@0"
I20260928 10:55:01.974762 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:01.993955 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:01.999189 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:01.999550 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:02.000711 21765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:02.002199 21765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:02.002278 21765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:02.002421 21765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:02.002478 21765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:02.042335 21765 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:35385
I20260928 10:55:02.042356 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:35385 every 8 connection(s)
I20260928 10:55:02.043812 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:55:02.044344 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 21765
I20260928 10:55:02.044586 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:55:02.045324 6030 external_mini_cluster.cc:1428] Setting key 36154398734c06c21422be5814409d70
Sep 28 10:55:02 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:55:02.073457 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.047668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:52959 (local address 127.5.227.190:36149)
0928 10:55:02.048207 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.048214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.049650 (+ 1436us) server_negotiation.cc:407] Connection header received
0928 10:55:02.053105 (+ 3455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.053110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.053190 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.053308 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:02.055859 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.056844 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.059007 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.059814 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.060109 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.064721 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:02.064730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:02.064739 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:02.064793 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:02.069037 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.070200 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.070208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.070214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:02.070385 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.071076 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.071084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.071090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:02.071560 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:02.071778 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.072730 (+ 952us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.073011 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":287,"threads_started":1}
I20260928 10:55:02.074414 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.048539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:52959)
0928 10:55:02.049438 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:02.049492 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:02.052662 (+ 3170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:02.053628 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:02.053644 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:02.054560 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:55:02.055508 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.055539 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.057086 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.057095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.058737 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.058753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.060157 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.060164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.060418 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.061284 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:55:02.061311 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:55:02.064387 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:55:02.069288 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:02.069299 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:02.069316 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:02.069912 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:02.070626 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:02.070631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:02.070634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:02.070932 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:02.071824 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:55:02.071832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:55:02.072262 (+ 430us) client_negotiation.cc:769] Sending connection context
0928 10:55:02.072581 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 10:55:02.073037 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":329,"threads_started":1}
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:55:02.076331 21893 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36149
I20260928 10:55:02.076746 21893 heartbeater.cc:464] Registering TS with master...
I20260928 10:55:02.077764 21893 heartbeater.cc:511] Master 127.5.227.190:36149 requested a full tablet report, sending...
I20260928 10:55:02.079922 21570 ts_manager.cc:194] Registered new tserver with Master: 06ad30337d914ff0a00965e2704fa6df (127.5.227.130:35385)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
I20260928 10:55:02.081105 21570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:52959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:55:02.116649 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36149
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:55:02.477656 21759 heartbeater.cc:503] Master 127.5.227.190:36149 was elected leader, sending a full tablet report...
W20260928 10:55:02.479856 21899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:02.480314 21899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:02.480509 21899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:02.488484 21899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:02.488621 21899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:02.488799 21899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:55:02.501021 21899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:02.502636 21899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:02.504168 21899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:02.519125 21905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:02.519090 21907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:02.519044 21904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:02.520562 21899 server_base.cc:1034] running on GCE node
I20260928 10:55:02.521667 21899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:02.522948 21899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:02.524210 21899 hybrid_clock.cc:648] HybridClock initialized: now 1790592902524141 us; error 66 us; skew 500 ppm
I20260928 10:55:02.524616 21899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:02 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592902, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:02.528374 21899 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:55:02.530303 21899 webserver.cc:460] Webserver started at http://127.5.227.131:44259/ using document root <none> and password file <none>
I20260928 10:55:02.531170 21899 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:02.531340 21899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:02.531697 21899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:55:02.534930 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "485d8eef0736485a80295176ee2b967f"
format_stamp: "Formatted at 2026-09-28 10:55:02 on dist-test-slave-8w8g"
server_key: "90fd2b6279ad3350284ae763442e9503"
server_key_iv: "fcfe21bd00039a85783c4dcac9c44844"
server_key_version: "encryptionkey@0"
I20260928 10:55:02.535938 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "485d8eef0736485a80295176ee2b967f"
format_stamp: "Formatted at 2026-09-28 10:55:02 on dist-test-slave-8w8g"
server_key: "90fd2b6279ad3350284ae763442e9503"
server_key_iv: "fcfe21bd00039a85783c4dcac9c44844"
server_key_version: "encryptionkey@0"
I20260928 10:55:02.542315 21899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:55:02.546761 21914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:02.548585 21899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:55:02.548786 21899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "485d8eef0736485a80295176ee2b967f"
format_stamp: "Formatted at 2026-09-28 10:55:02 on dist-test-slave-8w8g"
server_key: "90fd2b6279ad3350284ae763442e9503"
server_key_iv: "fcfe21bd00039a85783c4dcac9c44844"
server_key_version: "encryptionkey@0"
I20260928 10:55:02.549048 21899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:02.568835 21899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:02.573794 21899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:02.574133 21899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:02.575269 21899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:02.576810 21899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:02.576917 21899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:02.577071 21899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:02.577144 21899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:02.614334 21899 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:45653
I20260928 10:55:02.614393 22026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:45653 every 8 connection(s)
I20260928 10:55:02.615891 21899 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:55:02.625176 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 21899
I20260928 10:55:02.625389 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:55:02.625876 6030 external_mini_cluster.cc:1428] Setting key bad701485387197a0260cd496e04bf29
Sep 28 10:55:02 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592902, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:55:02.638378 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.618670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:41123 (local address 127.5.227.190:36149)
0928 10:55:02.619147 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.619151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.620187 (+ 1036us) server_negotiation.cc:407] Connection header received
0928 10:55:02.622188 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.622192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.622283 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.622387 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:02.624126 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.624952 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.626936 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.627473 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.627630 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.631783 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:02.631793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:02.631803 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:02.631842 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:02.634871 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.635817 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.635821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.635824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:02.635962 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.636454 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.636458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.636460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:02.636763 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:02.636964 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.637717 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.637973 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":223,"threads_started":1}
I20260928 10:55:02.639546 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.619219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:41123)
0928 10:55:02.620013 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:02.620061 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:02.621784 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:02.622650 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:02.622659 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:02.623188 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:55:02.623888 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.623910 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.625170 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.625174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.626780 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.626795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.627749 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.627757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.628010 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.628811 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:55:02.628835 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:55:02.631462 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:55:02.635120 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:02.635129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:02.635145 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:02.635630 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:02.636089 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:02.636094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:02.636097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:02.636311 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:02.636920 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:55:02.636938 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:55:02.637323 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 10:55:02.637613 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 10:55:02.638206 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":292,"threads_started":1}
I20260928 10:55:02.641383 22027 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36149
I20260928 10:55:02.641819 22027 heartbeater.cc:464] Registering TS with master...
I20260928 10:55:02.642843 22027 heartbeater.cc:511] Master 127.5.227.190:36149 requested a full tablet report, sending...
I20260928 10:55:02.645110 21569 ts_manager.cc:194] Registered new tserver with Master: 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653)
I20260928 10:55:02.646190 21569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:41123
I20260928 10:55:02.655627 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:55:02.677146 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.661402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.5.227.190:36149)
0928 10:55:02.661659 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.661666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.662271 (+ 605us) server_negotiation.cc:407] Connection header received
0928 10:55:02.662418 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.662423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.662515 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.662624 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:02.664362 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.665223 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.666670 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.667103 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.667257 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.670071 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:02.670077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:02.670080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:02.670120 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:02.674083 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.674989 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.674993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.674996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:02.675170 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.675629 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.675633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.675635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:02.676025 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:02.676222 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.676706 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.676904 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 10:55:02.682837 21569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:55:02.690716 21569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:55:02.719839 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.706871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.5.227.129:32825)
0928 10:55:02.709318 (+ 2447us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.709332 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.710321 (+ 989us) server_negotiation.cc:407] Connection header received
0928 10:55:02.710444 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.710452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.710686 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.710932 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:02.711710 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.713520 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.716705 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.718523 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.718702 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.719021 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.719271 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.719449 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2097,"thread_start_us":1913,"threads_started":1}
I20260928 10:55:02.719974 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.705826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:43568)
0928 10:55:02.709896 (+ 4070us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:02.709941 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:02.710164 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:02.710941 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:02.710950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:02.710985 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:02.711515 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.711526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.713694 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.713700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.716485 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.716503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.718739 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.718745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.718968 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.719041 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 10:55:02.719223 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 10:55:02.719449 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3913,"mutex_wait_us":165,"spinlock_wait_cycles":2304,"thread_start_us":4096,"threads_started":1}
I20260928 10:55:02.720800 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.705990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:45653 (local address 127.0.0.1:55408)
0928 10:55:02.709996 (+ 4006us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:02.710018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:02.710168 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:02.711165 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:02.711170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:02.711207 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:02.712042 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.712062 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.714208 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.714213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.717267 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.717284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.719837 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.719843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.720056 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.720101 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 10:55:02.720180 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 10:55:02.720335 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3860,"mutex_wait_us":382,"spinlock_wait_cycles":2688,"thread_start_us":3535,"threads_started":1}
I20260928 10:55:02.721339 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.705800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.5.227.130:35385)
0928 10:55:02.707595 (+ 1795us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.707610 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.709140 (+ 1530us) server_negotiation.cc:407] Connection header received
0928 10:55:02.710402 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.710410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.710691 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.710874 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:02.712290 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.713977 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.717480 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.719219 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.719363 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.719554 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.720683 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.721001 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1569,"thread_start_us":1398,"threads_started":1}
I20260928 10:55:02.721640 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.706388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.5.227.131:45653)
0928 10:55:02.708437 (+ 2049us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.708450 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.710322 (+ 1872us) server_negotiation.cc:407] Connection header received
0928 10:55:02.710416 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.710422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.710692 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.710874 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:02.712331 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.714039 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.717490 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.719219 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.719363 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.719604 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.721099 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.721281 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1813,"thread_start_us":1653,"threads_started":1}
I20260928 10:55:02.722185 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.705370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:35385 (local address 127.0.0.1:39462)
0928 10:55:02.708642 (+ 3272us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:02.708701 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:02.710100 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:02.711165 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:02.711170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:02.711219 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:02.712100 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.712114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.714207 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.714213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.717270 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.717284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.719824 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.719836 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.720437 (+ 601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.720479 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 10:55:02.721615 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0928 10:55:02.721774 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":322,"spinlock_wait_cycles":11648,"thread_start_us":637,"threads_started":1}
I20260928 10:55:02.725325 21828 tablet_service.cc:1467] Processing CreateTablet for tablet 0684b11472664f6f85c41e9d25a0db58 (DEFAULT_TABLE table=test-table [id=408352e0ad71426a967a9f6669bb027f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:55:02.725131 21694 tablet_service.cc:1467] Processing CreateTablet for tablet 0684b11472664f6f85c41e9d25a0db58 (DEFAULT_TABLE table=test-table [id=408352e0ad71426a967a9f6669bb027f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:55:02.725618 21962 tablet_service.cc:1467] Processing CreateTablet for tablet 0684b11472664f6f85c41e9d25a0db58 (DEFAULT_TABLE table=test-table [id=408352e0ad71426a967a9f6669bb027f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:55:02.726583 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684b11472664f6f85c41e9d25a0db58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:55:02.726725 21694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684b11472664f6f85c41e9d25a0db58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:55:02.727036 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684b11472664f6f85c41e9d25a0db58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:55:02.741740 22046 tablet_bootstrap.cc:492] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Bootstrap starting.
I20260928 10:55:02.744025 22047 tablet_bootstrap.cc:492] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Bootstrap starting.
I20260928 10:55:02.746094 22046 tablet_bootstrap.cc:654] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Neither blocks nor log segments found. Creating new log.
I20260928 10:55:02.746116 22048 tablet_bootstrap.cc:492] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Bootstrap starting.
I20260928 10:55:02.747728 22046 log.cc:824] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:02.748055 22047 tablet_bootstrap.cc:654] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Neither blocks nor log segments found. Creating new log.
I20260928 10:55:02.749867 22047 log.cc:824] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:02.750238 22048 tablet_bootstrap.cc:654] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Neither blocks nor log segments found. Creating new log.
I20260928 10:55:02.751370 22048 log.cc:824] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:02.752135 22046 tablet_bootstrap.cc:492] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: No bootstrap required, opened a new log
I20260928 10:55:02.752532 22046 ts_tablet_manager.cc:1397] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260928 10:55:02.754673 22048 tablet_bootstrap.cc:492] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: No bootstrap required, opened a new log
I20260928 10:55:02.754966 22048 ts_tablet_manager.cc:1397] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 10:55:02.762959 22047 tablet_bootstrap.cc:492] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: No bootstrap required, opened a new log
I20260928 10:55:02.763360 22047 ts_tablet_manager.cc:1397] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.010s
I20260928 10:55:02.763679 22048 raft_consensus.cc:348] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.764407 22048 raft_consensus.cc:374] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:55:02.764531 22048 raft_consensus.cc:729] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f74db17a0dd4281b8262486ff8b08c4, State: Initialized, Role: FOLLOWER
I20260928 10:55:02.765460 22048 consensus_queue.cc:260] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.767489 22048 ts_tablet_manager.cc:1428] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260928 10:55:02.767777 22046 raft_consensus.cc:348] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.768289 22046 raft_consensus.cc:374] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:55:02.768399 22046 raft_consensus.cc:729] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ad30337d914ff0a00965e2704fa6df, State: Initialized, Role: FOLLOWER
I20260928 10:55:02.769129 22046 consensus_queue.cc:260] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.770598 21893 heartbeater.cc:503] Master 127.5.227.190:36149 was elected leader, sending a full tablet report...
I20260928 10:55:02.771281 22046 ts_tablet_manager.cc:1428] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
I20260928 10:55:02.774612 22047 raft_consensus.cc:348] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.775121 22047 raft_consensus.cc:374] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:55:02.775210 22047 raft_consensus.cc:729] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485d8eef0736485a80295176ee2b967f, State: Initialized, Role: FOLLOWER
I20260928 10:55:02.775983 22047 consensus_queue.cc:260] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.777307 22027 heartbeater.cc:503] Master 127.5.227.190:36149 was elected leader, sending a full tablet report...
I20260928 10:55:02.777856 22047 ts_tablet_manager.cc:1428] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
W20260928 10:55:02.797259 21894 tablet.cc:2369] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:55:02.838045 22052 raft_consensus.cc:482] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:55:02.838414 22052 raft_consensus.cc:504] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.840360 22052 leader_election.cc:288] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06ad30337d914ff0a00965e2704fa6df (127.5.227.130:35385), 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653)
I20260928 10:55:02.849337 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.840981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:60311 (local address 127.5.227.131:45653)
0928 10:55:02.841173 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.841180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.843024 (+ 1844us) server_negotiation.cc:407] Connection header received
0928 10:55:02.843110 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.843114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.843211 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.843306 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:02.844090 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.844928 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.847112 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.848221 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.848413 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.848498 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.848869 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.849013 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:55:02.849351 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.840897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:59857 (local address 127.5.227.130:35385)
0928 10:55:02.841154 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.841161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.842758 (+ 1597us) server_negotiation.cc:407] Connection header received
0928 10:55:02.843021 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.843029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.843096 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.843185 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:02.843920 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.844694 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.847238 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.848382 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.848557 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.848633 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.848958 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.849088 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 10:55:02.849511 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.841154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:45653 (local address 127.5.227.129:60311)
0928 10:55:02.842781 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:02.842798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:02.842911 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:02.843381 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:02.843384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:02.843422 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:02.843857 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.843869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.845124 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.845128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.846887 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.846910 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.848474 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.848484 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.848678 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.848721 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:55:02.848806 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:55:02.848982 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":4,"thread_start_us":539,"threads_started":1}
I20260928 10:55:02.849522 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.841157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:35385 (local address 127.5.227.129:59857)
0928 10:55:02.842505 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:02.842522 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:02.842784 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:02.843232 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:02.843239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:02.843270 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:02.843670 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.843683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.844872 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.844878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.847046 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:02.847064 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.848624 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.848629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:02.848757 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.848802 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 10:55:02.848900 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 10:55:02.849031 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"mutex_wait_us":120,"spinlock_wait_cycles":16640,"thread_start_us":973,"threads_started":1}
I20260928 10:55:02.850559 21848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684b11472664f6f85c41e9d25a0db58" candidate_uuid: "5f74db17a0dd4281b8262486ff8b08c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ad30337d914ff0a00965e2704fa6df" is_pre_election: true
I20260928 10:55:02.850903 21982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684b11472664f6f85c41e9d25a0db58" candidate_uuid: "5f74db17a0dd4281b8262486ff8b08c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485d8eef0736485a80295176ee2b967f" is_pre_election: true
I20260928 10:55:02.851210 21848 raft_consensus.cc:2454] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f74db17a0dd4281b8262486ff8b08c4 in term 0.
I20260928 10:55:02.851471 21982 raft_consensus.cc:2454] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f74db17a0dd4281b8262486ff8b08c4 in term 0.
I20260928 10:55:02.852257 21649 leader_election.cc:302] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ad30337d914ff0a00965e2704fa6df, 5f74db17a0dd4281b8262486ff8b08c4; no voters:
I20260928 10:55:02.852924 22052 raft_consensus.cc:2781] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:55:02.853103 22052 raft_consensus.cc:482] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:55:02.853214 22052 raft_consensus.cc:3037] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:55:02.855641 22052 raft_consensus.cc:504] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
I20260928 10:55:02.856451 22052 leader_election.cc:288] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [CANDIDATE]: Term 1 election: Requested vote from peers 06ad30337d914ff0a00965e2704fa6df (127.5.227.130:35385), 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653)
I20260928 10:55:02.857162 21848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684b11472664f6f85c41e9d25a0db58" candidate_uuid: "5f74db17a0dd4281b8262486ff8b08c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ad30337d914ff0a00965e2704fa6df"
I20260928 10:55:02.857216 21982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684b11472664f6f85c41e9d25a0db58" candidate_uuid: "5f74db17a0dd4281b8262486ff8b08c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485d8eef0736485a80295176ee2b967f"
I20260928 10:55:02.857442 21982 raft_consensus.cc:3037] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:55:02.857578 21848 raft_consensus.cc:3037] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:55:02.861020 21848 raft_consensus.cc:2454] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f74db17a0dd4281b8262486ff8b08c4 in term 1.
I20260928 10:55:02.861021 21982 raft_consensus.cc:2454] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f74db17a0dd4281b8262486ff8b08c4 in term 1.
I20260928 10:55:02.861696 21648 leader_election.cc:302] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485d8eef0736485a80295176ee2b967f, 5f74db17a0dd4281b8262486ff8b08c4; no voters:
I20260928 10:55:02.862123 22052 raft_consensus.cc:2781] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:55:02.862686 22052 raft_consensus.cc:686] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5f74db17a0dd4281b8262486ff8b08c4, State: Running, Role: LEADER
I20260928 10:55:02.863384 22052 consensus_queue.cc:237] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } }
W20260928 10:55:02.867702 22028 tablet.cc:2369] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:55:02.868788 21569 catalog_manager.cc:5495] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f74db17a0dd4281b8262486ff8b08c4 (127.5.227.129). New cstate: current_term: 1 leader_uuid: "5f74db17a0dd4281b8262486ff8b08c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } health_report { overall_health: UNKNOWN } } }
I20260928 10:55:02.895378 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.887503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.5.227.129:32825)
0928 10:55:02.887771 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.887780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.888351 (+ 571us) server_negotiation.cc:407] Connection header received
0928 10:55:02.888502 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.888509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.888658 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.888839 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:55:02.889510 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.890490 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.892314 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.892814 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.892939 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.893858 (+ 919us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:55:02.894696 (+ 838us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:55:02.894822 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:02.895022 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 10:55:02.895147 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:55:02.904539 21848 raft_consensus.cc:1261] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 1 FOLLOWER]: Refusing update from remote peer 5f74db17a0dd4281b8262486ff8b08c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:55:02.904536 21982 raft_consensus.cc:1261] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Refusing update from remote peer 5f74db17a0dd4281b8262486ff8b08c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:55:02.905718 22057 consensus_queue.cc:1035] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:55:02.906347 22052 consensus_queue.cc:1035] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:55:02.921185 22060 mvcc.cc:205] Tried to move back new op lower bound from 7334268530286116864 to 7334268530133458944. Current Snapshot: MvccSnapshot[applied={T|T < 7334268530286116864}]
I20260928 10:55:02.922709 22062 mvcc.cc:205] Tried to move back new op lower bound from 7334268530286116864 to 7334268530133458944. Current Snapshot: MvccSnapshot[applied={T|T < 7334268530286116864}]
I20260928 10:55:02.927834 22061 mvcc.cc:205] Tried to move back new op lower bound from 7334268530286116864 to 7334268530133458944. Current Snapshot: MvccSnapshot[applied={T|T < 7334268530286116864}]
I20260928 10:55:02.942643 21569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:55:02.942960 21569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:55:02.947989 21569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Sending DeleteTablet for 3 replicas of tablet 0684b11472664f6f85c41e9d25a0db58
I20260928 10:55:02.949536 21828 tablet_service.cc:1514] Processing DeleteTablet for tablet 0684b11472664f6f85c41e9d25a0db58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:55:02 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:39462
I20260928 10:55:02.949713 21694 tablet_service.cc:1514] Processing DeleteTablet for tablet 0684b11472664f6f85c41e9d25a0db58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:55:02 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:43568
I20260928 10:55:02.950179 21962 tablet_service.cc:1514] Processing DeleteTablet for tablet 0684b11472664f6f85c41e9d25a0db58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:55:02 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:55408
I20260928 10:55:02.952030 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 21631
I20260928 10:55:02.965937 22072 tablet_replica.cc:330] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: stopping tablet replica
I20260928 10:55:02.966693 22072 raft_consensus.cc:2229] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:55:02.967600 22072 raft_consensus.cc:2258] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:55:02.968957 22071 tablet_replica.cc:330] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: stopping tablet replica
I20260928 10:55:02.969082 22070 tablet_replica.cc:330] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: stopping tablet replica
I20260928 10:55:02.969462 22070 raft_consensus.cc:2229] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:55:02.969483 22071 raft_consensus.cc:2229] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:55:02.970026 22070 raft_consensus.cc:2258] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:55:02.970402 22071 raft_consensus.cc:2258] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:55:02.972255 22072 ts_tablet_manager.cc:1905] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:55:02.972522 22070 ts_tablet_manager.cc:1905] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:55:02.973304 22071 ts_tablet_manager.cc:1905] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:55:02.980592 22072 ts_tablet_manager.cc:1918] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:55:02.980809 22072 log.cc:1196] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/0684b11472664f6f85c41e9d25a0db58
I20260928 10:55:02.981092 22070 ts_tablet_manager.cc:1918] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:55:02.981285 22070 log.cc:1196] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/0684b11472664f6f85c41e9d25a0db58
I20260928 10:55:02.981420 22072 ts_tablet_manager.cc:1939] T 0684b11472664f6f85c41e9d25a0db58 P 485d8eef0736485a80295176ee2b967f: Deleting consensus metadata
I20260928 10:55:02.981854 22070 ts_tablet_manager.cc:1939] T 0684b11472664f6f85c41e9d25a0db58 P 06ad30337d914ff0a00965e2704fa6df: Deleting consensus metadata
I20260928 10:55:02.984274 22071 ts_tablet_manager.cc:1918] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:55:02.984735 22071 log.cc:1196] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/0684b11472664f6f85c41e9d25a0db58
I20260928 10:55:02.985287 21555 catalog_manager.cc:4841] TS 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653): tablet 0684b11472664f6f85c41e9d25a0db58 (table test-table [id=408352e0ad71426a967a9f6669bb027f]) successfully deleted
I20260928 10:55:02.985392 22071 ts_tablet_manager.cc:1939] T 0684b11472664f6f85c41e9d25a0db58 P 5f74db17a0dd4281b8262486ff8b08c4: Deleting consensus metadata
I20260928 10:55:02.987644 21557 catalog_manager.cc:4841] TS 5f74db17a0dd4281b8262486ff8b08c4 (127.5.227.129:32825): tablet 0684b11472664f6f85c41e9d25a0db58 (table test-table [id=408352e0ad71426a967a9f6669bb027f]) successfully deleted
I20260928 10:55:02.987998 21556 catalog_manager.cc:4841] TS 06ad30337d914ff0a00965e2704fa6df (127.5.227.130:35385): tablet 0684b11472664f6f85c41e9d25a0db58 (table test-table [id=408352e0ad71426a967a9f6669bb027f]) successfully deleted
Sep 28 10:55:02 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:55:03.001780 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:02.954121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.5.227.129:32825)
0928 10:55:02.954373 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:02.954382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:02.959155 (+ 4773us) server_negotiation.cc:407] Connection header received
0928 10:55:02.960385 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:02.960399 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:02.960524 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:02.960630 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:02.966120 (+ 5490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.967574 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.983382 (+ 15808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:02.984289 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:02.984389 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:02.993151 (+ 8762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:02.993170 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:02.993191 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:02.993246 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:02.997215 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.998392 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.998401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.998405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:02.998602 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:02.999460 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:02.999464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:02.999467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:03.000075 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:03.000336 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:03.000856 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 10:55:03.001296 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:55:03.002703 21754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:03.110788 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 21631
I20260928 10:55:03.131699 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 21765
Sep 28 10:55:03 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:55:03.158300 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:03.133163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.5.227.130:35385)
0928 10:55:03.133351 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:03.133356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:03.133377 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:55:03.133520 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:03.133524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:03.133616 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:03.133710 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:03.135098 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:03.136269 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:03.137955 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:03.138357 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:03.138517 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:03.149763 (+ 11246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:03.149790 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:03.149808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:03.149858 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:03.154046 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:03.155198 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:03.155209 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:03.155212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:03.155385 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:03.156180 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:03.156184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:03.156187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:03.156641 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:03.156990 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:03.157579 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 10:55:03.157779 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:55:03.159271 21888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:03.259090 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 21765
I20260928 10:55:03.277488 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 21899
Sep 28 10:55:03 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:55:03.299497 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:03.282338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.5.227.131:45653)
0928 10:55:03.282614 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:03.282622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:03.282746 (+ 124us) server_negotiation.cc:407] Connection header received
0928 10:55:03.282986 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:03.282991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:03.283065 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:03.283185 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:03.284663 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:03.285391 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:03.286850 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:03.287272 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:03.287428 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:03.292892 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:03.292917 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:03.292951 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:03.293042 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:03.296397 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:03.297242 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:03.297250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:03.297253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:03.297403 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:03.297819 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:03.297825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:03.297830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:03.298241 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:03.298555 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:03.298892 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 10:55:03.299136 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 10:55:03.300450 22022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:03.393005 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 21899
I20260928 10:55:03.412387 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 21538
I20260928 10:55:03.413647 21600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:03.530474 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 21538
I20260928 10:55:23.554824 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36149
--webserver_interface=127.5.227.190
--webserver_port=39215
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:55:23.968703 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:23.969226 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:23.969424 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:23.978600 22078 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 10:55:23.978747 22078 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 10:55:23.978791 22078 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 10:55:23.978843 22078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:55:23.978904 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:23.978991 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:23.979041 22078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:55:23.979102 22078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:55:23.992129 22078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=39215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:23.994094 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:23.995651 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:24.024973 22085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:24.025084 22087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:24.025296 22078 server_base.cc:1034] running on GCE node
W20260928 10:55:24.025353 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:24.027027 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:24.029156 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:24.030648 22078 hybrid_clock.cc:648] HybridClock initialized: now 1790592924030561 us; error 55 us; skew 500 ppm
I20260928 10:55:24.031245 22078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:24 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592924, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:24.035694 22078 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:55:24.037983 22078 webserver.cc:460] Webserver started at http://127.5.227.190:39215/ using document root <none> and password file <none>
I20260928 10:55:24.038968 22078 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:24.039160 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:24.047690 22078 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:55:24.051872 22094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:24.054133 22078 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:55:24.054445 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "00059920d8414debbe150b9172f45d9d"
format_stamp: "Formatted at 2026-09-28 10:55:00 on dist-test-slave-8w8g"
server_key: "b16f8fd097dcd7ab90c1117dd2ee0965"
server_key_iv: "2d6a1479e2c2043dd63c6194caab8607"
server_key_version: "encryptionkey@0"
I20260928 10:55:24.055823 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:24.116374 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:24.122978 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:24.123432 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:24.150911 22078 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:36149
I20260928 10:55:24.150923 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:36149 every 8 connection(s)
I20260928 10:55:24.152791 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:55:24.159081 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22078
I20260928 10:55:24.160333 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:32825
--local_ip_for_outbound_sockets=127.5.227.129
--tserver_master_addrs=127.5.227.190:36149
--webserver_port=46025
--webserver_interface=127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:55:24.162803 22146 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:55:24.170171 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Bootstrap starting.
I20260928 10:55:24.195328 22146 log.cc:824] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:24.213794 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:55:24.214639 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Bootstrap complete.
I20260928 10:55:24.226711 22146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:24.227828 22146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00059920d8414debbe150b9172f45d9d, State: Initialized, Role: FOLLOWER
I20260928 10:55:24.228940 22146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:24.229269 22146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:55:24.229449 22146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:55:24.229664 22146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:55:24.234494 22146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:24.235206 22146 leader_election.cc:302] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00059920d8414debbe150b9172f45d9d; no voters:
I20260928 10:55:24.236007 22146 leader_election.cc:288] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:55:24.236173 22150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:55:24.237588 22150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 LEADER]: Becoming Leader. State: Replica: 00059920d8414debbe150b9172f45d9d, State: Running, Role: LEADER
I20260928 10:55:24.238407 22150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:24.238801 22146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:55:24.241703 22152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00059920d8414debbe150b9172f45d9d. Latest consensus state: current_term: 2 leader_uuid: "00059920d8414debbe150b9172f45d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } } }
I20260928 10:55:24.242182 22152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: This master's current role is: LEADER
I20260928 10:55:24.242841 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "00059920d8414debbe150b9172f45d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } } }
I20260928 10:55:24.243350 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: This master's current role is: LEADER
I20260928 10:55:24.253248 22156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:55:24.265056 22156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:55:24.267016 22156 catalog_manager.cc:1259] Loaded cluster ID: ee071bd6874f4f8ea2a9bb16548519d8
I20260928 10:55:24.267230 22156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:55:24.274303 22156 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:55:24.276810 22156 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Loaded TSK: 0
I20260928 10:55:24.301999 22156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Generated new TSK 1
I20260928 10:55:24.302420 22156 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:55:24.589144 22148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:24.589529 22148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:24.589696 22148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:24.597916 22148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:24.598100 22148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:24.598279 22148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:55:24.608899 22148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:32825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=46025
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:24.610342 22148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:24.611933 22148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:24.627705 22176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:24.627811 22173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:24.627753 22174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:24.634176 22148 server_base.cc:1034] running on GCE node
I20260928 10:55:24.635003 22148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:24.636350 22148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:24.637681 22148 hybrid_clock.cc:648] HybridClock initialized: now 1790592924637604 us; error 71 us; skew 500 ppm
I20260928 10:55:24.638083 22148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:24 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592924, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:24.642289 22148 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:55:24.644766 22148 webserver.cc:460] Webserver started at http://127.5.227.129:46025/ using document root <none> and password file <none>
I20260928 10:55:24.645936 22148 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:24.646140 22148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:24.655517 22148 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 10:55:24.659855 22183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:24.661959 22148 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:55:24.662194 22148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "5f74db17a0dd4281b8262486ff8b08c4"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "96c84353c636c1fe65c8207ced1e7525"
server_key_iv: "d7d99ac6732d2c5c8c4f6ba12f26b677"
server_key_version: "encryptionkey@0"
I20260928 10:55:24.663884 22148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:24.715123 22148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:24.722051 22148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:24.722432 22148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:24.724148 22148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:24.726261 22148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:24.726400 22148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:24.726511 22148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:24.726598 22148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:24.769455 22148 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:32825
I20260928 10:55:24.769603 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:32825 every 8 connection(s)
I20260928 10:55:24.771215 22148 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:55:24.776464 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22148
I20260928 10:55:24.777308 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:35385
--local_ip_for_outbound_sockets=127.5.227.130
--tserver_master_addrs=127.5.227.190:36149
--webserver_port=45419
--webserver_interface=127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:55:24 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592924, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:55:24.801508 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:24.774462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:50541 (local address 127.5.227.190:36149)
0928 10:55:24.775153 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:24.775166 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:24.776178 (+ 1012us) server_negotiation.cc:407] Connection header received
0928 10:55:24.778935 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:24.778961 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:24.779690 (+ 729us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:24.780253 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:24.782279 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:24.784378 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:24.786963 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:24.787713 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:24.788051 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:24.792221 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:24.792230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:24.792249 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:24.792296 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:24.795969 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:24.796970 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:24.796979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:24.796986 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:24.797188 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:24.797742 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:24.797748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:24.797752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:24.798310 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:24.798547 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:24.799457 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 10:55:24.799997 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":322,"threads_started":1}
I20260928 10:55:24.801545 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:24.774615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:50541)
0928 10:55:24.775780 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:24.775977 (+ 197us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:24.778173 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:24.780403 (+ 2230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:24.780438 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:24.781285 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:55:24.781982 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:24.782007 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:24.784673 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:24.784680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:24.786614 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:24.786635 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:24.788079 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:24.788088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:24.788422 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:24.789245 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:55:24.789272 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:55:24.791866 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:55:24.796201 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:24.796220 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:24.796235 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:24.796766 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:24.797355 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:24.797360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:24.797365 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:24.797603 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:24.798489 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:55:24.798501 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:55:24.798897 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 10:55:24.799218 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 10:55:24.799997 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":535,"threads_started":1}
I20260928 10:55:24.805395 22296 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36149
I20260928 10:55:24.805774 22296 heartbeater.cc:464] Registering TS with master...
I20260928 10:55:24.806715 22296 heartbeater.cc:511] Master 127.5.227.190:36149 requested a full tablet report, sending...
I20260928 10:55:24.809623 22111 ts_manager.cc:194] Registered new tserver with Master: 5f74db17a0dd4281b8262486ff8b08c4 (127.5.227.129:32825)
I20260928 10:55:24.812001 22111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:50541
W20260928 10:55:25.150918 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:25.151227 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:25.151335 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:25.161007 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:25.161136 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:25.161357 22300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:55:25.172580 22300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:35385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=45419
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:25.174048 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:25.175604 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:25.192313 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:25.192222 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:25.192343 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:25.198938 22300 server_base.cc:1034] running on GCE node
I20260928 10:55:25.199860 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:25.201195 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:25.202749 22300 hybrid_clock.cc:648] HybridClock initialized: now 1790592925202669 us; error 72 us; skew 500 ppm
I20260928 10:55:25.203164 22300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:25 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592925, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:25.207332 22300 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:55:25.209342 22300 webserver.cc:460] Webserver started at http://127.5.227.130:45419/ using document root <none> and password file <none>
I20260928 10:55:25.210180 22300 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:25.210345 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:25.217109 22300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:55:25.221086 22315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:25.222944 22300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:55:25.223167 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "06ad30337d914ff0a00965e2704fa6df"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "1c3f69b259662ce83e0894723e6ab75a"
server_key_iv: "cbadd871e5151ba4b26f72496120dacf"
server_key_version: "encryptionkey@0"
I20260928 10:55:25.224361 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:25.240268 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:25.246094 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:25.246440 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:25.247721 22300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:25.249468 22300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:25.249619 22300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:25.249773 22300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:25.249894 22300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:25.291045 22300 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:35385
I20260928 10:55:25.291160 22427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:35385 every 8 connection(s)
I20260928 10:55:25.292819 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:55:25.297503 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22300
I20260928 10:55:25.298399 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:45653
--local_ip_for_outbound_sockets=127.5.227.131
--tserver_master_addrs=127.5.227.190:36149
--webserver_port=44259
--webserver_interface=127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:55:25 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592925, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:55:25.323226 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.296153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:41645 (local address 127.5.227.190:36149)
0928 10:55:25.296362 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.296369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.298075 (+ 1706us) server_negotiation.cc:407] Connection header received
0928 10:55:25.301500 (+ 3425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.301506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.301620 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.301719 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:25.304363 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.305462 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.308020 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.308850 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.309192 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.313855 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:25.313866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:25.313874 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:25.313928 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:25.318234 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.319801 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.319811 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.319818 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.320039 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.320671 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.320681 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.320686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.321193 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:25.321436 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:25.322302 (+ 866us) server_negotiation.cc:299] Negotiation successful
0928 10:55:25.322687 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:55:25.323951 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.296807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:41645)
0928 10:55:25.297839 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:25.297901 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:25.301025 (+ 3124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:25.302083 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:25.302099 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:25.303028 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:55:25.304046 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.304080 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.305725 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.305735 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.307721 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.307803 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.309276 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.309282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.309528 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.310465 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:55:25.310493 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:55:25.313467 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:55:25.318824 (+ 5357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:25.318837 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:25.318863 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:25.319538 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:25.320219 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:25.320225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:25.320228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:25.320490 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:25.321384 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:55:25.321391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:55:25.321793 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 10:55:25.322098 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 10:55:25.322569 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":403,"threads_started":1}
I20260928 10:55:25.326133 22428 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36149
I20260928 10:55:25.326514 22428 heartbeater.cc:464] Registering TS with master...
I20260928 10:55:25.327323 22428 heartbeater.cc:511] Master 127.5.227.190:36149 requested a full tablet report, sending...
I20260928 10:55:25.329164 22111 ts_manager.cc:194] Registered new tserver with Master: 06ad30337d914ff0a00965e2704fa6df (127.5.227.130:35385)
I20260928 10:55:25.330263 22111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:41645
W20260928 10:55:25.679451 22431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:25.680040 22431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:25.680276 22431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:25.688220 22431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:25.688416 22431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:25.688594 22431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:55:25.699148 22431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:45653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=44259
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:25.700642 22431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:25.702093 22431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:25.717890 22436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:25.718091 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:25.717854 22437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:25.719079 22431 server_base.cc:1034] running on GCE node
I20260928 10:55:25.720163 22431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:25.721313 22431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:25.722610 22431 hybrid_clock.cc:648] HybridClock initialized: now 1790592925722546 us; error 68 us; skew 500 ppm
I20260928 10:55:25.723006 22431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:25 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592925, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:25.726768 22431 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:55:25.728932 22431 webserver.cc:460] Webserver started at http://127.5.227.131:44259/ using document root <none> and password file <none>
I20260928 10:55:25.729724 22431 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:25.729938 22431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:25.736552 22431 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:55:25.740234 22446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:25.742342 22431 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:55:25.742640 22431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "485d8eef0736485a80295176ee2b967f"
format_stamp: "Formatted at 2026-09-28 10:55:02 on dist-test-slave-8w8g"
server_key: "90fd2b6279ad3350284ae763442e9503"
server_key_iv: "fcfe21bd00039a85783c4dcac9c44844"
server_key_version: "encryptionkey@0"
I20260928 10:55:25.744115 22431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:25.761822 22431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:25.767160 22431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:25.767419 22431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:25.768747 22431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:25.770831 22431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:25.770949 22431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:25.771196 22431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:25.771518 22431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:25.812337 22431 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:45653
I20260928 10:55:25.812577 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:45653 every 8 connection(s)
I20260928 10:55:25.814430 22431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:55:25.817359 22296 heartbeater.cc:503] Master 127.5.227.190:36149 was elected leader, sending a full tablet report...
I20260928 10:55:25.823285 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22431
Sep 28 10:55:25 dist-test-slave-8w8g krb5kdc[21525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592925, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:55:25.842494 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.818341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:55471 (local address 127.5.227.190:36149)
0928 10:55:25.818580 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.818586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.820062 (+ 1476us) server_negotiation.cc:407] Connection header received
0928 10:55:25.822334 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.822339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.822542 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.822648 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:25.825018 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.826189 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.828937 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.829551 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.830079 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.835007 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:25.835012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:25.835015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:25.835055 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:25.838692 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.839781 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.839788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.839794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.839943 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.840528 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.840539 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.840541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.840861 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:25.841029 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:25.841861 (+ 832us) server_negotiation.cc:299] Negotiation successful
0928 10:55:25.842108 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 10:55:25.842829 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.824366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.5.227.190:36149)
0928 10:55:25.825869 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.825873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.825889 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:55:25.825943 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.825946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.826008 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.826145 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:25.828021 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.828815 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.830725 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.831250 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.831415 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.835722 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:25.835729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:25.835801 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:25.835861 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:25.839498 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.840658 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.840666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.840672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.840803 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.841238 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.841243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.841246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.841533 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:25.841837 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:25.842136 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 10:55:25.842374 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":1234,"threads_started":1}
I20260928 10:55:25.843710 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.818618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:55471)
0928 10:55:25.819707 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:25.819795 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:25.821980 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:25.823081 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:25.823095 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:25.823833 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:55:25.824704 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.824745 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.826409 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.826415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.828519 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.828536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.830210 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.830218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.830509 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.831345 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:55:25.831377 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:55:25.834676 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:55:25.838921 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:25.838936 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:25.838960 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:25.839585 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:25.840117 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:55:25.840124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:55:25.840127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:55:25.840377 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:55:25.841003 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:55:25.841012 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:55:25.841418 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 10:55:25.841705 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 10:55:25.842339 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":402,"threads_started":1}
I20260928 10:55:25.845413 22559 heartbeater.cc:347] Connected to a master server at 127.5.227.190:36149
I20260928 10:55:25.845767 22559 heartbeater.cc:464] Registering TS with master...
I20260928 10:55:25.846593 22559 heartbeater.cc:511] Master 127.5.227.190:36149 requested a full tablet report, sending...
I20260928 10:55:25.848706 22111 ts_manager.cc:194] Registered new tserver with Master: 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653)
I20260928 10:55:25.849685 22111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:55471
I20260928 10:55:25.858881 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:55:25.867415 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.860112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.5.227.190:36149)
0928 10:55:25.860433 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.860438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.860988 (+ 550us) server_negotiation.cc:407] Connection header received
0928 10:55:25.861162 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.861167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.861322 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.861481 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:55:25.862049 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.862829 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.865103 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.865734 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.866008 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.866624 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:55:25.866841 (+ 217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 10:55:25.867156 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54318: token expired
Metrics: {"server-negotiator.queue_time_us":124}
W20260928 10:55:25.867687 22562 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54318: token expired
W20260928 10:55:25.868276 22031 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.227.190:36149)
I20260928 10:55:25.886114 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.868946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.5.227.190:36149)
0928 10:55:25.869105 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.869109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.869125 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:55:25.869171 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.869174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.869232 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.869313 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:25.871397 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.872375 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.874578 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.875072 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.875194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.878635 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:25.878642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:25.878645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:25.878690 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:25.882278 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.883278 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.883284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.883287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.883500 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:25.884183 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:25.884188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:25.884190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:25.884580 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:25.884787 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:25.885303 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 10:55:25.885725 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:55:25.891422 22111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:55:25.900667 22111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:55:25.938282 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.916288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:35385 (local address 127.0.0.1:56140)
0928 10:55:25.917900 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:25.918064 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:25.918314 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:25.922329 (+ 4015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:25.922341 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:25.922389 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:25.922958 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.922977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.927431 (+ 4454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.927441 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.933000 (+ 5559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.933037 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.937067 (+ 4030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.937073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.937316 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.937400 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 10:55:25.937556 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 10:55:25.937844 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"thread_start_us":19695,"threads_started":1}
I20260928 10:55:25.938476 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.916914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.5.227.130:35385)
0928 10:55:25.919159 (+ 2245us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.919173 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.919216 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:55:25.921275 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.921289 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.921664 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.921941 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:25.923222 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.927153 (+ 3931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.934656 (+ 7503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.936813 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.936967 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.937201 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:25.937690 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 10:55:25.938124 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1926,"thread_start_us":1705,"threads_started":1}
I20260928 10:55:25.943323 22363 tablet_service.cc:1467] Processing CreateTablet for tablet a9755e71391d4be58e674d72afddf91e (DEFAULT_TABLE table=test-table [id=e1d68cba396e474184af980ba2fef890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:55:25.944845 22363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9755e71391d4be58e674d72afddf91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:55:25.945617 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.916812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:55936)
0928 10:55:25.933775 (+ 16963us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:25.933817 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:25.934764 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:25.935954 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:25.935960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:25.935990 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:25.936718 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.936758 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.939090 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.939095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.941931 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.941949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.944619 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.944626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.944845 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.944886 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 10:55:25.944984 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 10:55:25.945165 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16798,"mutex_wait_us":79,"thread_start_us":1170,"threads_started":1}
I20260928 10:55:25.945684 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.917706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.5.227.129:32825)
0928 10:55:25.934624 (+ 16918us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.934633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.934663 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:55:25.935053 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.935062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.935378 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.935606 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:25.936972 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.938886 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.942194 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.944021 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.944189 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.944441 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:25.945148 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 10:55:25.945355 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16722,"thread_start_us":16558,"threads_started":1}
I20260928 10:55:25.946256 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.916301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:45653 (local address 127.0.0.1:40572)
0928 10:55:25.936355 (+ 20054us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:25.936373 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:25.936537 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:25.937310 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:25.937315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:25.937344 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:25.937748 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.937761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.940353 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.940359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.942578 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:25.942595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.944891 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.944900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:25.945210 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.945253 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:55:25.945356 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 10:55:25.945516 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19890,"mutex_wait_us":907,"thread_start_us":16256,"threads_started":1}
I20260928 10:55:25.946959 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:25.916773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.5.227.131:45653)
0928 10:55:25.934010 (+ 17237us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:25.934025 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:25.936508 (+ 2483us) server_negotiation.cc:407] Connection header received
0928 10:55:25.936729 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:25.936736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:25.936998 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:25.937245 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:25.937962 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.940155 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.942808 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:25.944624 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:25.944768 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:25.945004 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:25.946046 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 10:55:25.946244 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16967,"thread_start_us":16771,"threads_started":1}
I20260928 10:55:25.950604 22231 tablet_service.cc:1467] Processing CreateTablet for tablet a9755e71391d4be58e674d72afddf91e (DEFAULT_TABLE table=test-table [id=e1d68cba396e474184af980ba2fef890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:55:25.951766 22494 tablet_service.cc:1467] Processing CreateTablet for tablet a9755e71391d4be58e674d72afddf91e (DEFAULT_TABLE table=test-table [id=e1d68cba396e474184af980ba2fef890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:55:25.953860 22231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9755e71391d4be58e674d72afddf91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:55:25.954088 22494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9755e71391d4be58e674d72afddf91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:55:25.963536 22575 tablet_bootstrap.cc:492] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Bootstrap starting.
I20260928 10:55:25.966641 22575 tablet_bootstrap.cc:654] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Neither blocks nor log segments found. Creating new log.
I20260928 10:55:25.967947 22575 log.cc:824] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:25.971452 22575 tablet_bootstrap.cc:492] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: No bootstrap required, opened a new log
I20260928 10:55:25.972025 22575 ts_tablet_manager.cc:1397] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 10:55:25.972254 22577 tablet_bootstrap.cc:492] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Bootstrap starting.
I20260928 10:55:25.973070 22578 tablet_bootstrap.cc:492] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Bootstrap starting.
I20260928 10:55:25.976562 22578 tablet_bootstrap.cc:654] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Neither blocks nor log segments found. Creating new log.
I20260928 10:55:25.976806 22577 tablet_bootstrap.cc:654] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Neither blocks nor log segments found. Creating new log.
I20260928 10:55:25.978014 22578 log.cc:824] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:25.978103 22577 log.cc:824] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:25.982333 22578 tablet_bootstrap.cc:492] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: No bootstrap required, opened a new log
I20260928 10:55:25.982981 22578 ts_tablet_manager.cc:1397] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 10:55:25.984711 22575 raft_consensus.cc:348] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:25.985212 22577 tablet_bootstrap.cc:492] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: No bootstrap required, opened a new log
I20260928 10:55:25.985227 22575 raft_consensus.cc:374] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:55:25.985538 22577 ts_tablet_manager.cc:1397] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260928 10:55:25.985699 22575 raft_consensus.cc:729] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ad30337d914ff0a00965e2704fa6df, State: Initialized, Role: FOLLOWER
I20260928 10:55:25.986533 22575 consensus_queue.cc:260] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:25.989303 22428 heartbeater.cc:503] Master 127.5.227.190:36149 was elected leader, sending a full tablet report...
I20260928 10:55:25.990459 22575 ts_tablet_manager.cc:1428] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260928 10:55:25.993039 22578 raft_consensus.cc:348] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:25.993619 22578 raft_consensus.cc:374] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:55:25.993721 22578 raft_consensus.cc:729] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485d8eef0736485a80295176ee2b967f, State: Initialized, Role: FOLLOWER
I20260928 10:55:25.994626 22578 consensus_queue.cc:260] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:25.996551 22559 heartbeater.cc:503] Master 127.5.227.190:36149 was elected leader, sending a full tablet report...
I20260928 10:55:25.998013 22578 ts_tablet_manager.cc:1428] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260928 10:55:26.001226 22577 raft_consensus.cc:348] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:26.002113 22577 raft_consensus.cc:374] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:55:26.002265 22577 raft_consensus.cc:729] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f74db17a0dd4281b8262486ff8b08c4, State: Initialized, Role: FOLLOWER
I20260928 10:55:26.003083 22577 consensus_queue.cc:260] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:26.005089 22577 ts_tablet_manager.cc:1428] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
W20260928 10:55:26.024335 22297 tablet.cc:2369] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:55:26.045614 22429 tablet.cc:2369] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:55:26.067189 22560 tablet.cc:2369] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:55:26.266288 22581 raft_consensus.cc:482] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:55:26.266582 22581 raft_consensus.cc:504] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:26.268494 22581 leader_election.cc:288] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653), 5f74db17a0dd4281b8262486ff8b08c4 (127.5.227.129:32825)
I20260928 10:55:26.278228 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.269088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:54299 (local address 127.5.227.129:32825)
0928 10:55:26.269513 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:26.269522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:26.270876 (+ 1354us) server_negotiation.cc:407] Connection header received
0928 10:55:26.271324 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:26.271328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:26.271517 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:26.271628 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:26.272372 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.273244 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.275350 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.276682 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.276861 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.276965 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:26.277554 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 10:55:26.277755 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 10:55:26.278297 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.269088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.5.227.130:54299)
0928 10:55:26.270684 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.270717 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:26.270957 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:26.271698 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:26.271702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:26.271776 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:26.272203 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:26.272215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.273409 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.273416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:26.275138 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:26.275154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.277135 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.277142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:26.277321 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.277375 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 10:55:26.277480 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 10:55:26.277661 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"mutex_wait_us":2,"thread_start_us":913,"threads_started":1}
I20260928 10:55:26.278796 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.269071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:46983 (local address 127.5.227.131:45653)
0928 10:55:26.269263 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:26.269267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:26.271239 (+ 1972us) server_negotiation.cc:407] Connection header received
0928 10:55:26.271515 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:26.271521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:26.271610 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:26.271697 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:55:26.272749 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.273760 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.276036 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.277280 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.277461 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.277539 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:26.278169 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 10:55:26.278523 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:55:26.278939 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.269876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:45653 (local address 127.5.227.130:46983)
0928 10:55:26.270948 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.270966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:55:26.271313 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:55:26.271869 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:55:26.271873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:55:26.271912 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:55:26.272482 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:26.272498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.274012 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.274016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:26.275825 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:55:26.275844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.277536 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.277543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:55:26.277742 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.277779 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 10:55:26.278087 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 10:55:26.278253 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":27,"thread_start_us":603,"threads_started":1}
I20260928 10:55:26.279471 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9755e71391d4be58e674d72afddf91e" candidate_uuid: "06ad30337d914ff0a00965e2704fa6df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f74db17a0dd4281b8262486ff8b08c4" is_pre_election: true
I20260928 10:55:26.279991 22514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9755e71391d4be58e674d72afddf91e" candidate_uuid: "06ad30337d914ff0a00965e2704fa6df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485d8eef0736485a80295176ee2b967f" is_pre_election: true
I20260928 10:55:26.280207 22251 raft_consensus.cc:2454] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ad30337d914ff0a00965e2704fa6df in term 0.
I20260928 10:55:26.280556 22514 raft_consensus.cc:2454] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06ad30337d914ff0a00965e2704fa6df in term 0.
I20260928 10:55:26.281172 22319 leader_election.cc:302] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ad30337d914ff0a00965e2704fa6df, 5f74db17a0dd4281b8262486ff8b08c4; no voters:
I20260928 10:55:26.281672 22581 raft_consensus.cc:2781] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:55:26.281929 22581 raft_consensus.cc:482] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:55:26.282064 22581 raft_consensus.cc:3037] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:55:26.284582 22581 raft_consensus.cc:504] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:26.285362 22581 leader_election.cc:288] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [CANDIDATE]: Term 1 election: Requested vote from peers 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653), 5f74db17a0dd4281b8262486ff8b08c4 (127.5.227.129:32825)
I20260928 10:55:26.286456 22514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9755e71391d4be58e674d72afddf91e" candidate_uuid: "06ad30337d914ff0a00965e2704fa6df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485d8eef0736485a80295176ee2b967f"
I20260928 10:55:26.286454 22251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9755e71391d4be58e674d72afddf91e" candidate_uuid: "06ad30337d914ff0a00965e2704fa6df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f74db17a0dd4281b8262486ff8b08c4"
I20260928 10:55:26.286988 22514 raft_consensus.cc:3037] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:55:26.286983 22251 raft_consensus.cc:3037] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:55:26.291481 22251 raft_consensus.cc:2454] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ad30337d914ff0a00965e2704fa6df in term 1.
I20260928 10:55:26.291481 22514 raft_consensus.cc:2454] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06ad30337d914ff0a00965e2704fa6df in term 1.
I20260928 10:55:26.292210 22317 leader_election.cc:302] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06ad30337d914ff0a00965e2704fa6df, 485d8eef0736485a80295176ee2b967f, 5f74db17a0dd4281b8262486ff8b08c4; no voters:
I20260928 10:55:26.292932 22581 raft_consensus.cc:2781] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:55:26.293584 22581 raft_consensus.cc:686] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 1 LEADER]: Becoming Leader. State: Replica: 06ad30337d914ff0a00965e2704fa6df, State: Running, Role: LEADER
I20260928 10:55:26.294124 22581 consensus_queue.cc:237] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } }
I20260928 10:55:26.299722 22111 catalog_manager.cc:5495] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df reported cstate change: term changed from 0 to 1, leader changed from <none> to 06ad30337d914ff0a00965e2704fa6df (127.5.227.130). New cstate: current_term: 1 leader_uuid: "06ad30337d914ff0a00965e2704fa6df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06ad30337d914ff0a00965e2704fa6df" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 35385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 } health_report { overall_health: UNKNOWN } } }
I20260928 10:55:26.400166 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.393413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.5.227.130:35385)
0928 10:55:26.393667 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:26.393675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:26.394134 (+ 459us) server_negotiation.cc:407] Connection header received
0928 10:55:26.394430 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:26.394434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:26.394520 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:26.394694 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:55:26.395198 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.395986 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.397370 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.397790 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.397949 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.398499 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:55:26.399249 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:55:26.399429 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:26.399621 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 10:55:26.399721 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:55:26.411705 22251 raft_consensus.cc:1261] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 FOLLOWER]: Refusing update from remote peer 06ad30337d914ff0a00965e2704fa6df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:55:26.412003 22514 raft_consensus.cc:1261] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Refusing update from remote peer 06ad30337d914ff0a00965e2704fa6df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:55:26.413646 22586 consensus_queue.cc:1035] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f74db17a0dd4281b8262486ff8b08c4" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 32825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:55:26.414089 22581 consensus_queue.cc:1035] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d8eef0736485a80295176ee2b967f" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:55:26.429534 22590 mvcc.cc:205] Tried to move back new op lower bound from 7334268626565824512 to 7334268626106355712. Current Snapshot: MvccSnapshot[applied={T|T < 7334268626565824512}]
I20260928 10:55:26.447486 22111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:55:26.448100 22111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:55:26.453122 22111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Sending DeleteTablet for 3 replicas of tablet a9755e71391d4be58e674d72afddf91e
I20260928 10:55:26.455376 22363 tablet_service.cc:1514] Processing DeleteTablet for tablet a9755e71391d4be58e674d72afddf91e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:55:26 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:56140
I20260928 10:55:26.455463 22494 tablet_service.cc:1514] Processing DeleteTablet for tablet a9755e71391d4be58e674d72afddf91e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:55:26 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:40572
I20260928 10:55:26.455874 22231 tablet_service.cc:1514] Processing DeleteTablet for tablet a9755e71391d4be58e674d72afddf91e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:55:26 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:55936
I20260928 10:55:26.456467 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 22148
I20260928 10:55:26.466678 22601 tablet_replica.cc:330] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: stopping tablet replica
I20260928 10:55:26.467628 22601 raft_consensus.cc:2229] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:55:26.468629 22600 tablet_replica.cc:330] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: stopping tablet replica
I20260928 10:55:26.470778 22600 raft_consensus.cc:2229] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:55:26.469704 22601 raft_consensus.cc:2258] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:55:26.472425 22600 raft_consensus.cc:2258] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:55:26.475878 22601 ts_tablet_manager.cc:1905] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:55:26.477108 22602 tablet_replica.cc:330] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: stopping tablet replica
I20260928 10:55:26.479312 22600 ts_tablet_manager.cc:1905] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:55:26.480014 22602 raft_consensus.cc:2229] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:55:26.480924 22602 raft_consensus.cc:2258] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:55:26.486053 22601 ts_tablet_manager.cc:1918] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:55:26.486119 22602 ts_tablet_manager.cc:1905] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:55:26.486291 22601 log.cc:1196] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/a9755e71391d4be58e674d72afddf91e
I20260928 10:55:26.486922 22601 ts_tablet_manager.cc:1939] T a9755e71391d4be58e674d72afddf91e P 06ad30337d914ff0a00965e2704fa6df: Deleting consensus metadata
I20260928 10:55:26.488998 22097 catalog_manager.cc:4841] TS 06ad30337d914ff0a00965e2704fa6df (127.5.227.130:35385): tablet a9755e71391d4be58e674d72afddf91e (table test-table [id=e1d68cba396e474184af980ba2fef890]) successfully deleted
I20260928 10:55:26.492973 22600 ts_tablet_manager.cc:1918] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:55:26.493342 22600 log.cc:1196] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/a9755e71391d4be58e674d72afddf91e
I20260928 10:55:26.494053 22600 ts_tablet_manager.cc:1939] T a9755e71391d4be58e674d72afddf91e P 485d8eef0736485a80295176ee2b967f: Deleting consensus metadata
I20260928 10:55:26.496176 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.458015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.5.227.129:32825)
0928 10:55:26.458225 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:26.458234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:26.473466 (+ 15232us) server_negotiation.cc:407] Connection header received
0928 10:55:26.474244 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:26.474253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:26.474354 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:26.474451 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:26.477285 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.478944 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.481486 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.482179 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.482301 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.486597 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:26.486621 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:26.486640 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:26.486698 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:26.491190 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:26.492397 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:26.492411 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:26.492416 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:26.492686 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:26.493468 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:26.493475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:26.493478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:26.494108 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:26.494547 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:26.495343 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 10:55:26.495675 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:55:26.497483 22096 catalog_manager.cc:4841] TS 485d8eef0736485a80295176ee2b967f (127.5.227.131:45653): tablet a9755e71391d4be58e674d72afddf91e (table test-table [id=e1d68cba396e474184af980ba2fef890]) successfully deleted
I20260928 10:55:26.497886 22291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:26.595925 22602 ts_tablet_manager.cc:1918] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:55:26.596253 22602 log.cc:1196] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/a9755e71391d4be58e674d72afddf91e
I20260928 10:55:26.596624 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 22148
W20260928 10:55:26.622067 22098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.227.129:32825 (error 108)
I20260928 10:55:26.622874 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 22300
I20260928 10:55:26.624655 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.623497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:55956)
0928 10:55:26.624247 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.624399 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":460,"threads_started":1}
W20260928 10:55:26.625092 22098 catalog_manager.cc:4584] TS 5f74db17a0dd4281b8262486ff8b08c4 (127.5.227.129:32825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9755e71391d4be58e674d72afddf91e: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
I20260928 10:55:26.641916 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.623692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.5.227.130:35385)
0928 10:55:26.624043 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:26.624050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:26.624603 (+ 553us) server_negotiation.cc:407] Connection header received
0928 10:55:26.624686 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:26.624691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:26.624781 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:26.625041 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:26.626710 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.627572 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.629324 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.629854 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.630003 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.633499 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:26.633516 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:26.633533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:26.633578 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:26.638098 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:26.639152 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:26.639160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:26.639163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:26.639353 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:26.639933 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:26.639937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:26.639940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:26.640482 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:26.640900 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:26.641215 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 10:55:26.641471 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 10:55:26.642680 22423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:26.679713 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.679089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:55968)
0928 10:55:26.679391 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.679524 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 10:55:26.681754 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.681093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:55976)
0928 10:55:26.681426 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.681536 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260928 10:55:26.743068 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.742226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:55988)
0928 10:55:26.742512 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.742666 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 10:55:26.744601 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.744083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:55992)
0928 10:55:26.744298 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.744437 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 10:55:26.791489 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 22300
I20260928 10:55:26.812849 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 22431
I20260928 10:55:26.825802 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.825239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:55996)
0928 10:55:26.825490 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.825605 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 10:55:26.827491 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.826821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:56004)
0928 10:55:26.827190 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:26.827296 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 10:55:26.833007 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:26.813785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.5.227.131:45653)
0928 10:55:26.814561 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 10:55:26.814567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:55:26.814586 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:55:26.814645 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:55:26.814650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:55:26.814723 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:55:26.814856 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:55:26.816726 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.818019 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.820195 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:55:26.820965 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:55:26.821119 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:55:26.825141 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:55:26.825154 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:55:26.825171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:55:26.825207 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:55:26.829412 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:26.830332 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:26.830340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:26.830343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:26.830495 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:55:26.830932 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:55:26.830939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:55:26.830947 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:55:26.831345 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:55:26.831690 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 10:55:26.832191 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 10:55:26.832575 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":514,"threads_started":1}
I20260928 10:55:26.833750 22554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:26.932148 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 22431
I20260928 10:55:26.949827 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 22078
I20260928 10:55:26.951382 22141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:55:27.069269 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 22078
I20260928 10:55:27.069367 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:27.068718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:32825 (local address 127.0.0.1:56020)
0928 10:55:27.068944 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:27.069102 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 10:55:47.093050 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36149
--webserver_interface=127.5.227.190
--webserver_port=39215
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:55:47.492815 22611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:47.493503 22611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:47.493706 22611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:47.502219 22611 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 10:55:47.502408 22611 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 10:55:47.502506 22611 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 10:55:47.502614 22611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:55:47.502707 22611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:47.502833 22611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:47.502919 22611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:55:47.502998 22611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:55:47.514196 22611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=39215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:47.515798 22611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:47.517506 22611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:47.532495 22617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:47.532593 22619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:47.532629 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:47.533481 22611 server_base.cc:1034] running on GCE node
I20260928 10:55:47.534358 22611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:47.536026 22611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:47.537318 22611 hybrid_clock.cc:648] HybridClock initialized: now 1790592947537246 us; error 33 us; skew 500 ppm
I20260928 10:55:47.537771 22611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:47 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592947, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:47.548807 22611 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:55:47.550590 22611 webserver.cc:460] Webserver started at http://127.5.227.190:39215/ using document root <none> and password file <none>
I20260928 10:55:47.551374 22611 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:47.551486 22611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:47.558678 22611 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:55:47.562489 22626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:47.564636 22611 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:55:47.564860 22611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "00059920d8414debbe150b9172f45d9d"
format_stamp: "Formatted at 2026-09-28 10:55:00 on dist-test-slave-8w8g"
server_key: "b16f8fd097dcd7ab90c1117dd2ee0965"
server_key_iv: "2d6a1479e2c2043dd63c6194caab8607"
server_key_version: "encryptionkey@0"
I20260928 10:55:47.565938 22611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:47.586566 22611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:47.592382 22611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:47.592760 22611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:47.618242 22611 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:36149
I20260928 10:55:47.618371 22677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:36149 every 8 connection(s)
I20260928 10:55:47.620592 22611 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:55:47.624913 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22611
I20260928 10:55:47.625981 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:32825
--local_ip_for_outbound_sockets=127.5.227.129
--tserver_master_addrs=127.5.227.190:36149
--webserver_port=46025
--webserver_interface=127.5.227.129
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:55:47.630172 22678 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:55:47.636437 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Bootstrap starting.
I20260928 10:55:47.661923 22678 log.cc:824] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:55:47.684897 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:55:47.685660 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Bootstrap complete.
I20260928 10:55:47.696183 22678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:47.697142 22678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00059920d8414debbe150b9172f45d9d, State: Initialized, Role: FOLLOWER
I20260928 10:55:47.697928 22678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:47.698223 22678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:55:47.698377 22678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:55:47.698581 22678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 2 FOLLOWER]: Advancing to term 3
I20260928 10:55:47.705250 22678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:47.706130 22678 leader_election.cc:302] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00059920d8414debbe150b9172f45d9d; no voters:
I20260928 10:55:47.707032 22678 leader_election.cc:288] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 10:55:47.707190 22682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 3 FOLLOWER]: Leader election won for term 3
I20260928 10:55:47.708959 22682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [term 3 LEADER]: Becoming Leader. State: Replica: 00059920d8414debbe150b9172f45d9d, State: Running, Role: LEADER
I20260928 10:55:47.709755 22682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } }
I20260928 10:55:47.709971 22678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:55:47.714726 22683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "00059920d8414debbe150b9172f45d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } } }
I20260928 10:55:47.714731 22684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00059920d8414debbe150b9172f45d9d. Latest consensus state: current_term: 3 leader_uuid: "00059920d8414debbe150b9172f45d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00059920d8414debbe150b9172f45d9d" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 36149 } } }
I20260928 10:55:47.715775 22684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: This master's current role is: LEADER
I20260928 10:55:47.715659 22683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d [sys.catalog]: This master's current role is: LEADER
I20260928 10:55:47.717244 22689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:55:47.740463 22689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:55:47.742491 22689 catalog_manager.cc:1259] Loaded cluster ID: ee071bd6874f4f8ea2a9bb16548519d8
I20260928 10:55:47.743434 22689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:55:47.748917 22689 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:55:47.751034 22689 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Loaded TSK: 0
I20260928 10:55:47.751351 22689 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00059920d8414debbe150b9172f45d9d: Loaded TSK: 1
F20260928 10:55:47.752774 22689 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe1a681c9ad google::LogMessage::Fail() at ??:0
@ 0x7fe1a681f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe1a681c4de google::LogMessage::Flush() at ??:0
@ 0x7fe1a681da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe1aa5807d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe1aa56f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe1bc24b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe1bc24a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe1bc2d62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe1bc2d60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe1bc3205e9 std::function<>::operator()() at ??:0
@ 0x7fe1bc24ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe1bc24cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe1bc2d4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe1bc2d4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe1a9468345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe1a946b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe1a946b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe1a9446eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe1ac7206db start_thread at ??:0
@ 0x7fe1a56b661f clone at ??:0
W20260928 10:55:48.146737 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:48.147094 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:48.147298 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:48.156390 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:48.156558 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:48.156809 22680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:55:48.169301 22680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:32825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=46025
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:48.170930 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:48.172536 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:48.192721 22725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:48.192757 22726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:48.192951 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:48.194496 22680 server_base.cc:1034] running on GCE node
I20260928 10:55:48.195582 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:48.197422 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:48.201062 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790592948200895 us; error 156 us; skew 500 ppm
I20260928 10:55:48.201601 22680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:48 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592948, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:48.206660 22680 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:55:48.208976 22680 webserver.cc:460] Webserver started at http://127.5.227.129:46025/ using document root <none> and password file <none>
I20260928 10:55:48.209903 22680 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:48.210103 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:48.217131 22680 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:55:48.220762 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:48.222810 22680 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:55:48.223094 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "5f74db17a0dd4281b8262486ff8b08c4"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "96c84353c636c1fe65c8207ced1e7525"
server_key_iv: "d7d99ac6732d2c5c8c4f6ba12f26b677"
server_key_version: "encryptionkey@0"
I20260928 10:55:48.224401 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:48.242517 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:48.249377 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:48.249759 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:48.251077 22680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:48.253536 22742 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 10:55:48.260370 22742 ts_tablet_manager.cc:1852] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 10:55:48.261519 22742 ts_tablet_manager.cc:1905] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:55:48.271225 22742 ts_tablet_manager.cc:1918] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:55:48.271608 22742 ts_tablet_manager.cc:1939] T a9755e71391d4be58e674d72afddf91e P 5f74db17a0dd4281b8262486ff8b08c4: Deleting consensus metadata
I20260928 10:55:48.272552 22680 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 10:55:48.272718 22680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.001s sys 0.000s
I20260928 10:55:48.272953 22680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:48.273051 22680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:48.315114 22680 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:32825
I20260928 10:55:48.315176 22848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:32825 every 8 connection(s)
I20260928 10:55:48.316730 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:55:48.322872 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:48.320286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:42735)
0928 10:55:48.321272 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:48.321549 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":441,"threads_started":1}
W20260928 10:55:48.324018 22849 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:36149 (0 consecutive failures): Network error: Failed to ping master at 127.5.227.190:36149: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
I20260928 10:55:48.324635 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22680
I20260928 10:55:48.325462 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:48.324913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:49207)
0928 10:55:48.325163 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:48.325323 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 10:55:48.325539 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:35385
--local_ip_for_outbound_sockets=127.5.227.130
--tserver_master_addrs=127.5.227.190:36149
--webserver_port=45419
--webserver_interface=127.5.227.130
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:55:48.327369 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:48.326670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:47961)
0928 10:55:48.327025 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:48.327146 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
W20260928 10:55:48.327800 22849 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:55:48.708581 22852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:48.708992 22852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:48.709190 22852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:48.718924 22852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:48.719152 22852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:48.719365 22852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:55:48.732788 22852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:35385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=45419
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:48.734266 22852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:48.735706 22852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:48.752401 22860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:48.752276 22858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:48.752427 22857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:48.752874 22852 server_base.cc:1034] running on GCE node
I20260928 10:55:48.753990 22852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:48.755203 22852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:48.756484 22852 hybrid_clock.cc:648] HybridClock initialized: now 1790592948756399 us; error 73 us; skew 500 ppm
I20260928 10:55:48.756920 22852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:48 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592948, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:48.760731 22852 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:55:48.762607 22852 webserver.cc:460] Webserver started at http://127.5.227.130:45419/ using document root <none> and password file <none>
I20260928 10:55:48.763366 22852 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:48.763485 22852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:48.770078 22852 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:55:48.773759 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:48.775722 22852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:55:48.776073 22852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "06ad30337d914ff0a00965e2704fa6df"
format_stamp: "Formatted at 2026-09-28 10:55:01 on dist-test-slave-8w8g"
server_key: "1c3f69b259662ce83e0894723e6ab75a"
server_key_iv: "cbadd871e5151ba4b26f72496120dacf"
server_key_version: "encryptionkey@0"
I20260928 10:55:48.777151 22852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:48.805073 22852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:48.810294 22852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:48.810627 22852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:48.811849 22852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:48.813405 22852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:48.813512 22852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:48.813647 22852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:48.813714 22852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:48.852481 22852 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:35385
I20260928 10:55:48.852528 22979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:35385 every 8 connection(s)
I20260928 10:55:48.854063 22852 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:55:48.856218 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22852
I20260928 10:55:48.857038 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:45653
--local_ip_for_outbound_sockets=127.5.227.131
--tserver_master_addrs=127.5.227.190:36149
--webserver_port=44259
--webserver_interface=127.5.227.131
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:55:48.860260 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:48.857864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:58529)
0928 10:55:48.858807 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:48.859083 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":380,"threads_started":1}
W20260928 10:55:48.861119 22980 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:36149 (0 consecutive failures): Network error: Failed to ping master at 127.5.227.190:36149: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
I20260928 10:55:48.862432 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:48.861803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:58911)
0928 10:55:48.862140 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:48.862254 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 10:55:48.864351 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:48.863100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:35501)
0928 10:55:48.863873 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:48.864058 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
W20260928 10:55:48.865077 22980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:55:49.223379 22984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:55:49.223899 22984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:55:49.224102 22984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:55:49.232393 22984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:55:49.232597 22984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:55:49.232808 22984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:55:49.243422 22984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:45653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=44259
--tserver_master_addrs=127.5.227.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:55:49.245077 22984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:55:49.246593 22984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:55:49.261942 22990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:49.262122 22989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:55:49.262244 22992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:55:49.263213 22984 server_base.cc:1034] running on GCE node
I20260928 10:55:49.264127 22984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:55:49.265347 22984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:55:49.266621 22984 hybrid_clock.cc:648] HybridClock initialized: now 1790592949266549 us; error 66 us; skew 500 ppm
I20260928 10:55:49.266983 22984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:55:49 dist-test-slave-8w8g krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592949, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:55:49.270979 22984 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:55:49.272855 22984 webserver.cc:460] Webserver started at http://127.5.227.131:44259/ using document root <none> and password file <none>
I20260928 10:55:49.273814 22984 fs_manager.cc:359] Metadata directory not provided
I20260928 10:55:49.273974 22984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:55:49.280507 22984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:55:49.284149 22999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:49.285792 22984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:55:49.286029 22984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "485d8eef0736485a80295176ee2b967f"
format_stamp: "Formatted at 2026-09-28 10:55:02 on dist-test-slave-8w8g"
server_key: "90fd2b6279ad3350284ae763442e9503"
server_key_iv: "fcfe21bd00039a85783c4dcac9c44844"
server_key_version: "encryptionkey@0"
I20260928 10:55:49.287155 22984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:55:49.311445 22984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:55:49.317137 22984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:55:49.317476 22984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:55:49.318558 22984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:55:49.320514 22984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:55:49.320655 22984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:49.320807 22984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:55:49.320899 22984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:55:49.329632 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:49.328572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:57135)
0928 10:55:49.329184 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:49.329319 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260928 10:55:49.363257 22984 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:45653
I20260928 10:55:49.363313 23112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:45653 every 8 connection(s)
I20260928 10:55:49.365262 22984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:55:49.368263 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 22984
I20260928 10:55:49.371392 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:49.368918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:38697)
0928 10:55:49.369819 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:49.370140 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":379,"threads_started":1}
W20260928 10:55:49.372452 23113 heartbeater.cc:644] Failed to heartbeat to 127.5.227.190:36149 (0 consecutive failures): Network error: Failed to ping master at 127.5.227.190:36149: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
I20260928 10:55:49.373874 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:49.373126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:40805)
0928 10:55:49.373648 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:49.373745 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
I20260928 10:55:49.375231 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:49.374683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:43941)
0928 10:55:49.374998 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:49.375088 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260928 10:55:49.375572 23113 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 10:55:49.867094 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:49.865909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:58133)
0928 10:55:49.866533 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:49.866650 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":340,"threads_started":1}
I20260928 10:55:50.331975 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:50.330921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:60467)
0928 10:55:50.331497 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:50.331604 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":349,"threads_started":1}
I20260928 10:55:50.377561 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:50.376451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:38681)
0928 10:55:50.377135 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:50.377235 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":402,"threads_started":1}
I20260928 10:55:50.869594 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:50.868747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:54503)
0928 10:55:50.869297 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:50.869416 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":313,"threads_started":1}
I20260928 10:55:51.333987 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:51.333145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:53005)
0928 10:55:51.333701 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:51.333805 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":282,"threads_started":1}
I20260928 10:55:51.379635 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:51.378724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:38371)
0928 10:55:51.379326 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:51.379461 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":313,"threads_started":1}
I20260928 10:55:51.871628 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:51.870787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:49477)
0928 10:55:51.871334 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:51.871431 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260928 10:55:52.336248 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:52.335264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:49467)
0928 10:55:52.335884 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:52.336024 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
I20260928 10:55:52.381637 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:52.380843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:57153)
0928 10:55:52.381349 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:52.381450 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":233,"threads_started":1}
I20260928 10:55:52.873653 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:52.872848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:44873)
0928 10:55:52.873382 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:52.873487 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":256,"threads_started":1}
I20260928 10:55:53.338337 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:53.337387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:56239)
0928 10:55:53.337998 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:53.338155 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":325,"threads_started":1}
I20260928 10:55:53.383883 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:53.382946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:60005)
0928 10:55:53.383516 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:53.383605 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":313,"threads_started":1}
I20260928 10:55:53.875895 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:53.874939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:43591)
0928 10:55:53.875551 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:53.875638 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":361,"threads_started":1}
I20260928 10:55:54.340253 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:54.339435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:39743)
0928 10:55:54.339970 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:54.340060 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":239,"threads_started":1}
I20260928 10:55:54.386059 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:54.384952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:56089)
0928 10:55:54.385610 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:54.385815 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":323,"threads_started":1}
I20260928 10:55:54.877840 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:54.876948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:53489)
0928 10:55:54.877528 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:54.877619 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":327,"threads_started":1}
I20260928 10:55:55.342465 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:55.341463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:35569)
0928 10:55:55.342103 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:55.342204 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":321,"threads_started":1}
I20260928 10:55:55.388037 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:55.387157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:48009)
0928 10:55:55.387651 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:55.387854 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":242,"threads_started":1}
I20260928 10:55:55.880119 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:55.879069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:42639)
0928 10:55:55.879624 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:55.879821 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":248,"threads_started":1}
I20260928 10:55:56.344784 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:56.343729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:37023)
0928 10:55:56.344479 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:56.344576 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":298,"threads_started":1}
I20260928 10:55:56.390374 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:56.389223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:56771)
0928 10:55:56.390054 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:56.390142 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":270,"threads_started":1}
I20260928 10:55:56.882395 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:56.881175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:53129)
0928 10:55:56.881903 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:56.881997 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":451,"threads_started":1}
I20260928 10:55:57.346897 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:57.345983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:48067)
0928 10:55:57.346541 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:57.346635 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":311,"threads_started":1}
I20260928 10:55:57.392588 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:57.391597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:48459)
0928 10:55:57.392227 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:57.392328 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":380,"threads_started":1}
I20260928 10:55:57.884966 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:57.884002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:33655)
0928 10:55:57.884600 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:57.884729 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":324,"threads_started":1}
I20260928 10:55:58.349154 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:58.348284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:38607)
0928 10:55:58.348870 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:58.348965 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":270,"threads_started":1}
I20260928 10:55:58.394555 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:58.393613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:59431)
0928 10:55:58.394262 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:58.394358 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":271,"threads_started":1}
I20260928 10:55:58.887044 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:58.886167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:44707)
0928 10:55:58.886707 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:58.886802 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":307,"threads_started":1}
I20260928 10:55:59.351923 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:59.350577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:58367)
0928 10:55:59.351395 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:59.351485 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":427,"threads_started":1}
I20260928 10:55:59.396848 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:59.395881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:54419)
0928 10:55:59.396428 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:59.396565 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":321,"threads_started":1}
I20260928 10:55:59.889279 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:55:59.888340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:38579)
0928 10:55:59.888952 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 10:55:59.889061 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":326,"threads_started":1}
I20260928 10:56:00.354307 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:00.353342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:48493)
0928 10:56:00.353958 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:00.354061 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":353,"threads_started":1}
I20260928 10:56:00.398886 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:00.398080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:48175)
0928 10:56:00.398565 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:00.398705 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":239,"threads_started":1}
I20260928 10:56:00.891813 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:00.890410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:55191)
0928 10:56:00.891440 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:00.891558 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":537,"threads_started":1}
I20260928 10:56:01.356799 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:01.355550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:33187)
0928 10:56:01.356342 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:01.356480 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":472,"threads_started":1}
I20260928 10:56:01.401001 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:01.400126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:56403)
0928 10:56:01.400715 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:01.400825 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":285,"threads_started":1}
I20260928 10:56:01.893721 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:01.892843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:60483)
0928 10:56:01.893402 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:01.893494 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":345,"threads_started":1}
I20260928 10:56:02.359117 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:02.358160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:57287)
0928 10:56:02.358774 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:02.358904 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":286,"threads_started":1}
I20260928 10:56:02.403295 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:02.402259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:50883)
0928 10:56:02.402917 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:02.403072 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260928 10:56:02.895967 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:02.894984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:35661)
0928 10:56:02.895563 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:02.895667 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
I20260928 10:56:03.361547 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:03.360551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:53589)
0928 10:56:03.361255 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:03.361377 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":427,"threads_started":1}
I20260928 10:56:03.405508 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:03.404528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:52519)
0928 10:56:03.405153 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:03.405296 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":353,"threads_started":1}
I20260928 10:56:03.898241 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:03.897308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.130:58127)
0928 10:56:03.897906 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:03.898031 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":287,"threads_started":1}
I20260928 10:56:04.363853 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:04.362836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.129:47947)
0928 10:56:04.363367 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:04.363458 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":264,"threads_started":1}
I20260928 10:56:04.372143 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 22680
I20260928 10:56:04.395931 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:04.373712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.5.227.129:32825)
0928 10:56:04.374510 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:04.374531 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:04.374586 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:56:04.376482 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:04.376494 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:04.376963 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:04.377284 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:04.378972 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:04.380439 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:04.382025 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:04.382956 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:04.383201 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:04.386862 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:04.386873 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:04.386892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:04.386950 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:04.390970 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:04.392433 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:04.392546 (+ 113us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:04.392559 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:04.392938 (+ 379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:04.393461 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:04.393465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:04.393467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:04.394214 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:04.394472 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:04.395058 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 10:56:04.395485 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":444,"threads_started":1}
I20260928 10:56:04.397607 22844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:04.407963 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:04.407033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:36149 (local address 127.5.227.131:41669)
0928 10:56:04.407584 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:04.407704 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.190:36149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":311,"threads_started":1}
I20260928 10:56:04.496321 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 22680
I20260928 10:56:04.514995 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 22852
I20260928 10:56:04.537420 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:04.516267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.5.227.130:35385)
0928 10:56:04.516844 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:04.516860 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:04.516902 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:56:04.518580 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:04.518592 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:04.519092 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:04.519505 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:04.521265 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:04.522700 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:04.524306 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:04.525283 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:04.525539 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:04.528818 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:04.528827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:04.528846 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:04.528896 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:04.532755 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:04.533856 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:04.533864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:04.533873 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:04.534144 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:04.534886 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:04.534892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:04.534901 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:04.535591 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:04.535996 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:04.536558 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 10:56:04.537085 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
I20260928 10:56:04.539027 22975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:04.631153 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 22852
I20260928 10:56:04.649680 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 22984
I20260928 10:56:04.671242 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:04.650962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.5.227.131:45653)
0928 10:56:04.651870 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:04.651892 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:04.651942 (+ 50us) server_negotiation.cc:407] Connection header received
0928 10:56:04.653883 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:04.653895 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:04.654392 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:04.654762 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:04.656241 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:04.657580 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:04.659004 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:04.659824 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:04.659965 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:04.662901 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:04.662910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:04.662935 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:04.662985 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:04.666782 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:04.667650 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:04.667658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:04.667667 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:04.667986 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:04.668464 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:04.668468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:04.668471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:04.669167 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:04.669407 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:04.670164 (+ 757us) server_negotiation.cc:299] Negotiation successful
0928 10:56:04.670829 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":401,"threads_started":1}
I20260928 10:56:04.673087 23108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:04.763705 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 22984
2026-09-28T10:56:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66211 ms)
[----------] 30 tests from SecurityITest (344990 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:04 dist-test-slave-8w8g krb5kdc[23170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:04 dist-test-slave-8w8g krb5kdc[23170](info): set up 2 sockets
Sep 28 10:56:04 dist-test-slave-8w8g krb5kdc[23170](info): commencing operation
krb5kdc: starting...
W20260928 10:56:05.928625 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:06 dist-test-slave-8w8g krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:56:06.230053 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37829
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:06.614157 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:06.614549 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:06.614732 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:06.622716 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:06.622887 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:06.622994 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:06.623100 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:06.623183 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:06.633872 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:06.635411 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:06.636976 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:06.651337 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:06.651320 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:06.651401 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:06.652405 23183 server_base.cc:1034] running on GCE node
I20260928 10:56:06.653555 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:06.655381 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:06.656876 23183 hybrid_clock.cc:648] HybridClock initialized: now 1790592966656808 us; error 103 us; skew 500 ppm
I20260928 10:56:06.657445 23183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:06 dist-test-slave-8w8g krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592966, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:06.662079 23183 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:56:06.664227 23183 webserver.cc:460] Webserver started at http://127.5.227.190:37307/ using document root <none> and password file <none>
I20260928 10:56:06.665046 23183 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:06.665207 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:06.665548 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:06.669786 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "1ebd77b685c2434cbfa31e4b73bf9116"
format_stamp: "Formatted at 2026-09-28 10:56:06 on dist-test-slave-8w8g"
server_key: "e48a3af97bf173b715cbf2e074812281"
server_key_iv: "bc6b03f004a510a9e059be4a7379357c"
server_key_version: "encryptionkey@0"
I20260928 10:56:06.671064 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "1ebd77b685c2434cbfa31e4b73bf9116"
format_stamp: "Formatted at 2026-09-28 10:56:06 on dist-test-slave-8w8g"
server_key: "e48a3af97bf173b715cbf2e074812281"
server_key_iv: "bc6b03f004a510a9e059be4a7379357c"
server_key_version: "encryptionkey@0"
I20260928 10:56:06.677750 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260928 10:56:06.682402 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:06.684942 23183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:56:06.685214 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "1ebd77b685c2434cbfa31e4b73bf9116"
format_stamp: "Formatted at 2026-09-28 10:56:06 on dist-test-slave-8w8g"
server_key: "e48a3af97bf173b715cbf2e074812281"
server_key_iv: "bc6b03f004a510a9e059be4a7379357c"
server_key_version: "encryptionkey@0"
I20260928 10:56:06.685464 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:06.735127 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:06.741253 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:06.741624 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:06.766000 23183 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:37829
I20260928 10:56:06.766068 23249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:37829 every 8 connection(s)
I20260928 10:56:06.767567 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:06.770210 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 23183
I20260928 10:56:06.770478 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:06.771462 6030 external_mini_cluster.cc:1428] Setting key cea010d351db599d3fe1d8ca5eab08ab
I20260928 10:56:06.773829 23250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:56:06 dist-test-slave-8w8g krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:06.789227 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116: Bootstrap starting.
I20260928 10:56:06.795760 23250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:06.797806 23250 log.cc:824] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:06.802047 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116: No bootstrap required, opened a new log
I20260928 10:56:06.802627 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:06.774264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.5.227.190:37829)
0928 10:56:06.775295 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:06.775310 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:06.775357 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:56:06.777374 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:06.777404 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:06.777907 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:06.778411 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:06.780254 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:06.781844 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:06.783400 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:06.784412 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:06.784681 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:06.790072 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:06.790087 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:06.790111 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:06.790306 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:06.796245 (+ 5939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:06.797428 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:06.797440 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:06.797451 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:06.797875 (+ 424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:06.798508 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:06.798517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:06.798524 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:06.799321 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:06.799779 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:06.800484 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 10:56:06.801057 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":450,"threads_started":1}
I20260928 10:56:06.812220 23250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37829 } }
I20260928 10:56:06.812672 23250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:06.812747 23250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ebd77b685c2434cbfa31e4b73bf9116, State: Initialized, Role: FOLLOWER
I20260928 10:56:06.813506 23250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37829 } }
I20260928 10:56:06.813861 23250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:06.814059 23250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:06.814280 23250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:06.816834 23250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37829 } }
I20260928 10:56:06.817404 23250 leader_election.cc:302] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ebd77b685c2434cbfa31e4b73bf9116; no voters:
I20260928 10:56:06.818100 23250 leader_election.cc:288] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:06.818205 23255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:06.818918 23255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [term 1 LEADER]: Becoming Leader. State: Replica: 1ebd77b685c2434cbfa31e4b73bf9116, State: Running, Role: LEADER
I20260928 10:56:06.819559 23255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37829 } }
I20260928 10:56:06.820751 23250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:06.823514 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ebd77b685c2434cbfa31e4b73bf9116. Latest consensus state: current_term: 1 leader_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37829 } } }
I20260928 10:56:06.824118 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:06.826628 23261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:06.826478 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebd77b685c2434cbfa31e4b73bf9116" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37829 } } }
I20260928 10:56:06.827041 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:06.833607 23261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:06.843546 23261 catalog_manager.cc:1347] Generated new cluster ID: 021f1385737f4f739e2fd063fe6e8c0f
I20260928 10:56:06.843888 23261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:06.868476 23261 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:06.870630 23261 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:06.881004 23261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116: Generated new TSK 0
I20260928 10:56:06.882294 23261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:56:06.964845 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37829
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:07.381435 23276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:07.381798 23276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:07.381955 23276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:07.391835 23276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:07.392071 23276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:07.392349 23276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:56:07.404603 23276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:07.406147 23276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:07.408330 23276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:07.425133 23284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:07.425261 23281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:07.425238 23282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:07.425884 23276 server_base.cc:1034] running on GCE node
I20260928 10:56:07.426668 23276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:07.427983 23276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:07.429319 23276 hybrid_clock.cc:648] HybridClock initialized: now 1790592967429259 us; error 39 us; skew 500 ppm
I20260928 10:56:07.429790 23276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:07 dist-test-slave-8w8g krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592967, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:07.441231 23276 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:56:07.444100 23276 webserver.cc:460] Webserver started at http://127.5.227.129:36575/ using document root <none> and password file <none>
I20260928 10:56:07.445007 23276 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:07.445204 23276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:07.445623 23276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:07.449260 23276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "4d0da56b400346e8ae8b2e560a587aa6"
format_stamp: "Formatted at 2026-09-28 10:56:07 on dist-test-slave-8w8g"
server_key: "32474e2dbdb99ac48470f05d30bb6030"
server_key_iv: "a7d943f75ca619ce8c1c57c654a1f993"
server_key_version: "encryptionkey@0"
I20260928 10:56:07.450126 23276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "4d0da56b400346e8ae8b2e560a587aa6"
format_stamp: "Formatted at 2026-09-28 10:56:07 on dist-test-slave-8w8g"
server_key: "32474e2dbdb99ac48470f05d30bb6030"
server_key_iv: "a7d943f75ca619ce8c1c57c654a1f993"
server_key_version: "encryptionkey@0"
I20260928 10:56:07.457726 23276 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 10:56:07.462302 23291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:07.464214 23276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:56:07.464396 23276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "4d0da56b400346e8ae8b2e560a587aa6"
format_stamp: "Formatted at 2026-09-28 10:56:07 on dist-test-slave-8w8g"
server_key: "32474e2dbdb99ac48470f05d30bb6030"
server_key_iv: "a7d943f75ca619ce8c1c57c654a1f993"
server_key_version: "encryptionkey@0"
I20260928 10:56:07.464625 23276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:07.485678 23276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:07.491142 23276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:07.491513 23276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:07.492633 23276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:07.494325 23276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:07.494406 23276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:07.494522 23276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:07.494570 23276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:07.535216 23276 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:41033
I20260928 10:56:07.535331 23403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:41033 every 8 connection(s)
I20260928 10:56:07.536675 23276 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:56:07.539525 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 23276
I20260928 10:56:07.539796 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:56:07.540692 6030 external_mini_cluster.cc:1428] Setting key 186d64079793b0eeae5ada771a914a1a
Sep 28 10:56:07 dist-test-slave-8w8g krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592967, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:07.566351 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:07.539642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:60845 (local address 127.5.227.190:37829)
0928 10:56:07.540298 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:07.540306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:07.541497 (+ 1191us) server_negotiation.cc:407] Connection header received
0928 10:56:07.543877 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:07.543882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:07.543954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:07.544058 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:07.546749 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:07.547606 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:07.550669 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:07.551229 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:07.551358 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:07.556904 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:07.556915 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:07.556925 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:07.556984 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:07.561379 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:07.562888 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:07.562898 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:07.562905 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:07.563100 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:07.563767 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:07.563775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:07.563782 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:07.564237 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:07.564420 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:07.565475 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0928 10:56:07.565796 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":358,"threads_started":1}
I20260928 10:56:07.567008 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:07.540326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37829 (local address 127.5.227.129:60845)
0928 10:56:07.541310 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:07.541367 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:07.543461 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:07.544379 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:07.544395 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:07.545521 (+ 1126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:07.546432 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:07.546469 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:07.547871 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:07.547879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:07.550310 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:07.550333 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:07.551532 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:07.551542 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:07.552000 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:07.553175 (+ 1175us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:07.553213 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:07.556551 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:07.561718 (+ 5167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:07.561734 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:07.561760 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:07.562666 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:07.563293 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:07.563298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:07.563301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:07.563560 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:07.564400 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:07.564407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:07.564996 (+ 589us) client_negotiation.cc:769] Sending connection context
0928 10:56:07.565287 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 10:56:07.565750 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":354,"threads_started":1}
I20260928 10:56:07.569270 23404 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37829
I20260928 10:56:07.569721 23404 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:07.570883 23404 heartbeater.cc:511] Master 127.5.227.190:37829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
I20260928 10:56:07.574338 23215 ts_manager.cc:194] Registered new tserver with Master: 4d0da56b400346e8ae8b2e560a587aa6 (127.5.227.129:41033)
I20260928 10:56:07.576592 23215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:60845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:56:07.612417 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37829
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:08.022459 23410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:08.022814 23410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:08.023002 23410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:08.033124 23410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:08.033320 23410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:08.033509 23410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:56:08.046944 23410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:08.048957 23410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:08.050805 23410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:08.066897 23418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:08.066970 23415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:08.066938 23416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:08.068082 23410 server_base.cc:1034] running on GCE node
I20260928 10:56:08.068807 23410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:08.070006 23410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:08.071303 23410 hybrid_clock.cc:648] HybridClock initialized: now 1790592968071258 us; error 47 us; skew 500 ppm
I20260928 10:56:08.071722 23410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:08 dist-test-slave-8w8g krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:08.075672 23410 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:56:08.077543 23410 webserver.cc:460] Webserver started at http://127.5.227.130:40229/ using document root <none> and password file <none>
I20260928 10:56:08.078341 23410 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:08.078464 23410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:08.078763 23410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:08.082367 23410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "5a9171811fd24705b5c010d19f1c62c2"
format_stamp: "Formatted at 2026-09-28 10:56:08 on dist-test-slave-8w8g"
server_key: "058cbcabc91da91e65d32366189d155e"
server_key_iv: "c66d9f5955d1dd48bc887066b872b1f9"
server_key_version: "encryptionkey@0"
I20260928 10:56:08.083285 23410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "5a9171811fd24705b5c010d19f1c62c2"
format_stamp: "Formatted at 2026-09-28 10:56:08 on dist-test-slave-8w8g"
server_key: "058cbcabc91da91e65d32366189d155e"
server_key_iv: "c66d9f5955d1dd48bc887066b872b1f9"
server_key_version: "encryptionkey@0"
I20260928 10:56:08.091154 23410 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 10:56:08.095702 23425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:08.097842 23410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:08.098086 23410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "5a9171811fd24705b5c010d19f1c62c2"
format_stamp: "Formatted at 2026-09-28 10:56:08 on dist-test-slave-8w8g"
server_key: "058cbcabc91da91e65d32366189d155e"
server_key_iv: "c66d9f5955d1dd48bc887066b872b1f9"
server_key_version: "encryptionkey@0"
I20260928 10:56:08.098335 23410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:08.116914 23410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:08.123854 23410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:08.124250 23410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:08.125679 23410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:08.127339 23410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:08.127492 23410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:08.127609 23410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:08.127681 23410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:08.175573 23410 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:41887
I20260928 10:56:08.175872 23537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:41887 every 8 connection(s)
I20260928 10:56:08.177090 23410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:56:08.183787 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 23410
I20260928 10:56:08.184001 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:56:08.184923 6030 external_mini_cluster.cc:1428] Setting key 2fa69681e33783344ff9094c32b73f74
Sep 28 10:56:08 dist-test-slave-8w8g krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:08.211055 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.182435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:36587 (local address 127.5.227.190:37829)
0928 10:56:08.183499 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.183508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.185072 (+ 1564us) server_negotiation.cc:407] Connection header received
0928 10:56:08.187450 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.187456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.187550 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.187647 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:08.190476 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.191371 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.194307 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.194890 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.195031 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.199993 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:08.200003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:08.200011 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:08.200071 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:08.205335 (+ 5264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:08.206901 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:08.206911 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:08.206917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:08.207130 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:08.207771 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:08.207779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:08.207786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:08.208221 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:08.208393 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.209801 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.210328 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":671,"threads_started":1}
I20260928 10:56:08.212517 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.183432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37829 (local address 127.5.227.130:36587)
0928 10:56:08.184814 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:08.184869 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:08.187035 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:08.188129 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:08.188144 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:08.189032 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:08.190056 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.190182 (+ 126us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.191583 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.191592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.194010 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.194039 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.195251 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.195263 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.195672 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.196678 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:08.196726 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:08.199465 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:08.205738 (+ 6273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:08.205754 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:08.205772 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:08.206696 (+ 924us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:08.207310 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:08.207314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:08.207317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:08.207562 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:08.208403 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:08.208410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:08.209121 (+ 711us) client_negotiation.cc:769] Sending connection context
0928 10:56:08.209437 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 10:56:08.210335 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":398,"threads_started":1}
I20260928 10:56:08.215315 23538 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37829
I20260928 10:56:08.215881 23538 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:08.216806 23538 heartbeater.cc:511] Master 127.5.227.190:37829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
I20260928 10:56:08.220144 23215 ts_manager.cc:194] Registered new tserver with Master: 5a9171811fd24705b5c010d19f1c62c2 (127.5.227.130:41887)
I20260928 10:56:08.221297 23215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:36587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:56:08.270592 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37829
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:56:08.581470 23404 heartbeater.cc:503] Master 127.5.227.190:37829 was elected leader, sending a full tablet report...
W20260928 10:56:08.655269 23544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:08.655681 23544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:08.655880 23544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:08.663887 23544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:08.664050 23544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:08.664237 23544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:56:08.674974 23544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:08.676530 23544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:08.677958 23544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:08.694892 23549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:08.694844 23552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:08.694939 23550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:08.701366 23544 server_base.cc:1034] running on GCE node
I20260928 10:56:08.702333 23544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:08.703794 23544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:08.705302 23544 hybrid_clock.cc:648] HybridClock initialized: now 1790592968705220 us; error 70 us; skew 500 ppm
I20260928 10:56:08.705752 23544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:08 dist-test-slave-8w8g krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:08.710057 23544 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:56:08.712633 23544 webserver.cc:460] Webserver started at http://127.5.227.131:43273/ using document root <none> and password file <none>
I20260928 10:56:08.713586 23544 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:08.713780 23544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:08.714149 23544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:08.717741 23544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "fbc65d12dc824e87a910cd5511aa3943"
format_stamp: "Formatted at 2026-09-28 10:56:08 on dist-test-slave-8w8g"
server_key: "b9cac67713b034c12a4557896cd960e8"
server_key_iv: "bea0cac8782b0ab2b19596ac7a96f4c3"
server_key_version: "encryptionkey@0"
I20260928 10:56:08.718767 23544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "fbc65d12dc824e87a910cd5511aa3943"
format_stamp: "Formatted at 2026-09-28 10:56:08 on dist-test-slave-8w8g"
server_key: "b9cac67713b034c12a4557896cd960e8"
server_key_iv: "bea0cac8782b0ab2b19596ac7a96f4c3"
server_key_version: "encryptionkey@0"
I20260928 10:56:08.725852 23544 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260928 10:56:08.731148 23559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:08.732983 23544 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:08.733219 23544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "fbc65d12dc824e87a910cd5511aa3943"
format_stamp: "Formatted at 2026-09-28 10:56:08 on dist-test-slave-8w8g"
server_key: "b9cac67713b034c12a4557896cd960e8"
server_key_iv: "bea0cac8782b0ab2b19596ac7a96f4c3"
server_key_version: "encryptionkey@0"
I20260928 10:56:08.733464 23544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:08.756184 23544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:08.761387 23544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:08.761778 23544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:08.763005 23544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:08.764875 23544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:08.765048 23544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:08.765255 23544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:08.765352 23544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:08.808398 23544 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:39477
I20260928 10:56:08.808509 23671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:39477 every 8 connection(s)
I20260928 10:56:08.810250 23544 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:56:08.812319 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 23544
I20260928 10:56:08.812572 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:56:08.813335 6030 external_mini_cluster.cc:1428] Setting key 93e0ec5d399a1eeb006f7da346f34ac2
Sep 28 10:56:08 dist-test-slave-8w8g krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:08.837859 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.813714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:48435 (local address 127.5.227.190:37829)
0928 10:56:08.814367 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.814372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.815776 (+ 1404us) server_negotiation.cc:407] Connection header received
0928 10:56:08.818358 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.818362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.818436 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.818548 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:08.820491 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.821512 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.823569 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.824135 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.824269 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.829105 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:08.829111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:08.829116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:08.829175 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:08.833190 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:08.834438 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:08.834447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:08.834450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:08.834668 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:08.835519 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:08.835523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:08.835526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:08.836084 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:08.836292 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.837182 (+ 890us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.837422 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":253,"threads_started":1}
I20260928 10:56:08.838781 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.814370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37829 (local address 127.5.227.131:48435)
0928 10:56:08.815534 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:08.815605 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:08.818076 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:08.818868 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:08.818879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:08.819466 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:08.820278 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.820301 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.821803 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.821808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.823387 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.823402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.824531 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.824537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.824775 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.825636 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:08.825668 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:08.828558 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:08.833382 (+ 4824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:08.833391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:08.833416 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:08.834223 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:08.834983 (+ 760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:08.834987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:08.834990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:08.835260 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:08.836303 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:08.836310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:08.836737 (+ 427us) client_negotiation.cc:769] Sending connection context
0928 10:56:08.836995 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 10:56:08.837573 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":607,"threads_started":1}
I20260928 10:56:08.840459 23672 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37829
I20260928 10:56:08.840744 23672 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:08.841639 23672 heartbeater.cc:511] Master 127.5.227.190:37829 requested a full tablet report, sending...
I20260928 10:56:08.843721 23215 ts_manager.cc:194] Registered new tserver with Master: fbc65d12dc824e87a910cd5511aa3943 (127.5.227.131:39477)
I20260928 10:56:08.844529 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:56:08.845085 23215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:48435
I20260928 10:56:08.865651 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.850423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.5.227.190:37829)
0928 10:56:08.850635 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.850641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.851272 (+ 631us) server_negotiation.cc:407] Connection header received
0928 10:56:08.851420 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.851425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.851517 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.851636 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:08.853499 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.854412 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.856129 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.856586 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.856695 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.859433 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:08.859438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:08.859442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:08.859483 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:08.862349 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:08.863297 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:08.863301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:08.863304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:08.863441 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:08.863999 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:08.864003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:08.864005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:08.864401 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:08.864609 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.865216 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.865409 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:56:08.871023 23215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:56:08.879184 23215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:56:08.907267 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.895546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:39477 (local address 127.0.0.1:59380)
0928 10:56:08.896403 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:08.896478 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:08.896685 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:08.899111 (+ 2426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:08.899115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:08.899148 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:08.899688 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.899705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.901936 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.901941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.904235 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.904250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.906136 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.906143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.906360 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.906458 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 10:56:08.906546 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 10:56:08.906816 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":134,"thread_start_us":412,"threads_started":1}
I20260928 10:56:08.907333 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.895927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.5.227.131:39477)
0928 10:56:08.898467 (+ 2540us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.898479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.898525 (+ 46us) server_negotiation.cc:407] Connection header received
0928 10:56:08.898617 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.898624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.898850 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.899975 (+ 1125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:08.900276 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.901778 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.904425 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.905910 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.906453 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.906710 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.906797 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.906980 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2350,"thread_start_us":2195,"threads_started":1}
I20260928 10:56:08.907928 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.896062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:41887 (local address 127.0.0.1:45466)
0928 10:56:08.897714 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:08.897732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:08.897881 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:08.898660 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:08.898671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:08.898718 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:08.899166 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.899187 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.901256 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.901263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.903953 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.903969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.906045 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.906052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.906295 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.906396 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 10:56:08.906591 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 10:56:08.906817 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"mutex_wait_us":87,"thread_start_us":1402,"threads_started":1}
I20260928 10:56:08.908597 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.896579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.5.227.130:41887)
0928 10:56:08.897264 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.897276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.897844 (+ 568us) server_negotiation.cc:407] Connection header received
0928 10:56:08.898057 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.898069 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.898356 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.898591 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:08.899372 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.901090 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.904177 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.905817 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.905989 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.906247 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.906941 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.908207 (+ 1266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":421,"threads_started":1}
I20260928 10:56:08.909293 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.897643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.5.227.129:41033)
0928 10:56:08.899251 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.899262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.900160 (+ 898us) server_negotiation.cc:407] Connection header received
0928 10:56:08.900266 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.900273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.900511 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.900706 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:08.901533 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.902688 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.905619 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.907064 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.907204 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.907430 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.908912 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.909064 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1425,"thread_start_us":1249,"threads_started":1}
I20260928 10:56:08.909826 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.898031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:41033 (local address 127.0.0.1:50110)
0928 10:56:08.899886 (+ 1855us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:08.899909 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:08.900026 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:08.900917 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:08.900921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:08.900950 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:08.901339 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.901353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.902843 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.902848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.905392 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.905405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.907601 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.907608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.907834 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.907964 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 10:56:08.908067 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 10:56:08.908215 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"mutex_wait_us":1,"thread_start_us":1536,"threads_started":1}
I20260928 10:56:08.913121 23607 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb92757584d41f484f30379610bc86f (DEFAULT_TABLE table=test-table [id=6a39bde5299a49448a60bc1121755a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:08.913195 23473 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb92757584d41f484f30379610bc86f (DEFAULT_TABLE table=test-table [id=6a39bde5299a49448a60bc1121755a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:08.914475 23607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb92757584d41f484f30379610bc86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:08.914650 23473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb92757584d41f484f30379610bc86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:08.914580 23339 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb92757584d41f484f30379610bc86f (DEFAULT_TABLE table=test-table [id=6a39bde5299a49448a60bc1121755a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:08.916164 23339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb92757584d41f484f30379610bc86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:08.927626 23691 tablet_bootstrap.cc:492] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Bootstrap starting.
I20260928 10:56:08.931385 23692 tablet_bootstrap.cc:492] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Bootstrap starting.
I20260928 10:56:08.933135 23691 tablet_bootstrap.cc:654] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:08.933735 23693 tablet_bootstrap.cc:492] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Bootstrap starting.
I20260928 10:56:08.935242 23691 log.cc:824] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:08.936131 23692 tablet_bootstrap.cc:654] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:08.937647 23692 log.cc:824] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:08.938144 23693 tablet_bootstrap.cc:654] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:08.939251 23691 tablet_bootstrap.cc:492] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: No bootstrap required, opened a new log
I20260928 10:56:08.939553 23693 log.cc:824] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:08.939611 23691 ts_tablet_manager.cc:1397] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260928 10:56:08.941267 23692 tablet_bootstrap.cc:492] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: No bootstrap required, opened a new log
I20260928 10:56:08.941568 23692 ts_tablet_manager.cc:1397] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 10:56:08.942915 23693 tablet_bootstrap.cc:492] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: No bootstrap required, opened a new log
I20260928 10:56:08.943223 23693 ts_tablet_manager.cc:1397] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 10:56:08.952488 23691 raft_consensus.cc:348] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.953045 23691 raft_consensus.cc:374] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:08.952488 23693 raft_consensus.cc:348] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.953142 23691 raft_consensus.cc:729] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc65d12dc824e87a910cd5511aa3943, State: Initialized, Role: FOLLOWER
I20260928 10:56:08.953279 23693 raft_consensus.cc:374] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:08.953379 23693 raft_consensus.cc:729] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0da56b400346e8ae8b2e560a587aa6, State: Initialized, Role: FOLLOWER
I20260928 10:56:08.953401 23692 raft_consensus.cc:348] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.954026 23692 raft_consensus.cc:374] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:08.954120 23692 raft_consensus.cc:729] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9171811fd24705b5c010d19f1c62c2, State: Initialized, Role: FOLLOWER
I20260928 10:56:08.954022 23691 consensus_queue.cc:260] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.954226 23693 consensus_queue.cc:260] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.954764 23692 consensus_queue.cc:260] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.959389 23693 ts_tablet_manager.cc:1428] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260928 10:56:08.962216 23691 ts_tablet_manager.cc:1428] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Time spent starting tablet: real 0.022s user 0.018s sys 0.004s
I20260928 10:56:08.962214 23538 heartbeater.cc:503] Master 127.5.227.190:37829 was elected leader, sending a full tablet report...
I20260928 10:56:08.962716 23672 heartbeater.cc:503] Master 127.5.227.190:37829 was elected leader, sending a full tablet report...
I20260928 10:56:08.963340 23692 ts_tablet_manager.cc:1428] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Time spent starting tablet: real 0.022s user 0.010s sys 0.011s
I20260928 10:56:08.969297 23699 raft_consensus.cc:482] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:56:08.969775 23699 raft_consensus.cc:504] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.972208 23699 leader_election.cc:288] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc65d12dc824e87a910cd5511aa3943 (127.5.227.131:39477), 4d0da56b400346e8ae8b2e560a587aa6 (127.5.227.129:41033)
I20260928 10:56:08.982542 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.974228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:54695 (local address 127.5.227.129:41033)
0928 10:56:08.974438 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.974445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.974754 (+ 309us) server_negotiation.cc:407] Connection header received
0928 10:56:08.974857 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.974861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.974925 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.975000 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:08.975704 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.976584 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.979929 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.981060 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.981235 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.981298 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.982090 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.982250 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:56:08.982667 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.973815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:41033 (local address 127.5.227.130:54695)
0928 10:56:08.974546 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:08.974574 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:08.974735 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:08.975041 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:08.975044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:08.975064 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:08.975521 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.975530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.976731 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.976737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.979529 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.979552 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.981448 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.981461 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.981869 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.981908 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:56:08.982121 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 10:56:08.982272 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":145,"thread_start_us":406,"threads_started":1}
I20260928 10:56:08.984272 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.973028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:49721 (local address 127.5.227.131:39477)
0928 10:56:08.973353 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:08.973363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:08.974061 (+ 698us) server_negotiation.cc:407] Connection header received
0928 10:56:08.974394 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:08.974400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:08.974485 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:08.974556 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:08.975523 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.976685 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.981401 (+ 4716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.982476 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.982899 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.983017 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:08.983882 (+ 865us) server_negotiation.cc:299] Negotiation successful
0928 10:56:08.984019 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 10:56:08.984325 23359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb92757584d41f484f30379610bc86f" candidate_uuid: "5a9171811fd24705b5c010d19f1c62c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0da56b400346e8ae8b2e560a587aa6" is_pre_election: true
I20260928 10:56:08.984946 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:08.972840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:39477 (local address 127.5.227.130:49721)
0928 10:56:08.973925 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:08.973956 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:08.974174 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:08.974802 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:08.974807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:08.974834 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:08.975306 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.975319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.976796 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.976801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.979529 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:08.979552 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:08.983177 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:08.983184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:08.983417 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:08.983456 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:56:08.983851 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 10:56:08.984030 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":200,"thread_start_us":511,"threads_started":1}
I20260928 10:56:08.985440 23359 raft_consensus.cc:2454] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a9171811fd24705b5c010d19f1c62c2 in term 0.
I20260928 10:56:08.986330 23627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb92757584d41f484f30379610bc86f" candidate_uuid: "5a9171811fd24705b5c010d19f1c62c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc65d12dc824e87a910cd5511aa3943" is_pre_election: true
I20260928 10:56:08.986539 23428 leader_election.cc:302] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0da56b400346e8ae8b2e560a587aa6, 5a9171811fd24705b5c010d19f1c62c2; no voters:
I20260928 10:56:08.986932 23627 raft_consensus.cc:2454] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a9171811fd24705b5c010d19f1c62c2 in term 0.
I20260928 10:56:08.987011 23699 raft_consensus.cc:2781] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:56:08.987170 23699 raft_consensus.cc:482] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:56:08.987283 23699 raft_consensus.cc:3037] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:08.990411 23699 raft_consensus.cc:504] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:08.991394 23699 leader_election.cc:288] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [CANDIDATE]: Term 1 election: Requested vote from peers fbc65d12dc824e87a910cd5511aa3943 (127.5.227.131:39477), 4d0da56b400346e8ae8b2e560a587aa6 (127.5.227.129:41033)
I20260928 10:56:08.992216 23359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb92757584d41f484f30379610bc86f" candidate_uuid: "5a9171811fd24705b5c010d19f1c62c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0da56b400346e8ae8b2e560a587aa6"
I20260928 10:56:08.992220 23627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb92757584d41f484f30379610bc86f" candidate_uuid: "5a9171811fd24705b5c010d19f1c62c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc65d12dc824e87a910cd5511aa3943"
I20260928 10:56:08.992537 23627 raft_consensus.cc:3037] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:08.992554 23359 raft_consensus.cc:3037] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:08.995129 23627 raft_consensus.cc:2454] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a9171811fd24705b5c010d19f1c62c2 in term 1.
I20260928 10:56:08.995661 23428 leader_election.cc:302] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a9171811fd24705b5c010d19f1c62c2, fbc65d12dc824e87a910cd5511aa3943; no voters:
I20260928 10:56:08.995918 23359 raft_consensus.cc:2454] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a9171811fd24705b5c010d19f1c62c2 in term 1.
I20260928 10:56:08.996140 23699 raft_consensus.cc:2781] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:08.996699 23699 raft_consensus.cc:686] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 1 LEADER]: Becoming Leader. State: Replica: 5a9171811fd24705b5c010d19f1c62c2, State: Running, Role: LEADER
I20260928 10:56:08.997319 23699 consensus_queue.cc:237] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } }
I20260928 10:56:09.003895 23214 catalog_manager.cc:5495] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a9171811fd24705b5c010d19f1c62c2 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "5a9171811fd24705b5c010d19f1c62c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a9171811fd24705b5c010d19f1c62c2" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41887 } health_report { overall_health: HEALTHY } } }
W20260928 10:56:09.039422 23405 tablet.cc:2369] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:09.047951 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:09.041533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.5.227.130:41887)
0928 10:56:09.041806 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:09.041815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:09.041920 (+ 105us) server_negotiation.cc:407] Connection header received
0928 10:56:09.042241 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:09.042246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:09.042329 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:09.042500 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:56:09.042925 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.043681 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.045350 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.045822 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.045974 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:09.046653 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:56:09.047253 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:56:09.047416 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:09.047500 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 10:56:09.047595 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 10:56:09.058688 23627 raft_consensus.cc:1261] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 1 FOLLOWER]: Refusing update from remote peer 5a9171811fd24705b5c010d19f1c62c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:09.058676 23359 raft_consensus.cc:1261] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 1 FOLLOWER]: Refusing update from remote peer 5a9171811fd24705b5c010d19f1c62c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:09.059921 23702 consensus_queue.cc:1035] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc65d12dc824e87a910cd5511aa3943" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 39477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:09.060406 23699 consensus_queue.cc:1035] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0da56b400346e8ae8b2e560a587aa6" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:56:09.062321 23673 tablet.cc:2369] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:09.098780 23213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:09.099159 23213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:09.105038 23213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ebd77b685c2434cbfa31e4b73bf9116: Sending DeleteTablet for 3 replicas of tablet 5cb92757584d41f484f30379610bc86f
I20260928 10:56:09.106807 23607 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb92757584d41f484f30379610bc86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:09 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:59380
I20260928 10:56:09.107177 23339 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb92757584d41f484f30379610bc86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:09 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:50110
I20260928 10:56:09.107396 23473 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb92757584d41f484f30379610bc86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:09 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:45466
I20260928 10:56:09.110453 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 23276
I20260928 10:56:09.119362 23717 tablet_replica.cc:330] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: stopping tablet replica
I20260928 10:56:09.119536 23716 tablet_replica.cc:330] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: stopping tablet replica
I20260928 10:56:09.119972 23717 raft_consensus.cc:2229] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:56:09.119973 23716 raft_consensus.cc:2229] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:09.120603 23716 raft_consensus.cc:2258] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:09.120921 23717 raft_consensus.cc:2258] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:09.123309 23716 ts_tablet_manager.cc:1905] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:09.123317 23717 ts_tablet_manager.cc:1905] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:09.133163 23716 ts_tablet_manager.cc:1918] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:09.133162 23717 ts_tablet_manager.cc:1918] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:09.133406 23717 log.cc:1196] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/5cb92757584d41f484f30379610bc86f
I20260928 10:56:09.133535 23716 log.cc:1196] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/5cb92757584d41f484f30379610bc86f
I20260928 10:56:09.134315 23717 ts_tablet_manager.cc:1939] T 5cb92757584d41f484f30379610bc86f P 5a9171811fd24705b5c010d19f1c62c2: Deleting consensus metadata
I20260928 10:56:09.134426 23716 ts_tablet_manager.cc:1939] T 5cb92757584d41f484f30379610bc86f P 4d0da56b400346e8ae8b2e560a587aa6: Deleting consensus metadata
I20260928 10:56:09.136543 23202 catalog_manager.cc:4841] TS 5a9171811fd24705b5c010d19f1c62c2 (127.5.227.130:41887): tablet 5cb92757584d41f484f30379610bc86f (table test-table [id=6a39bde5299a49448a60bc1121755a95]) successfully deleted
I20260928 10:56:09.136576 23201 catalog_manager.cc:4841] TS 4d0da56b400346e8ae8b2e560a587aa6 (127.5.227.129:41033): tablet 5cb92757584d41f484f30379610bc86f (table test-table [id=6a39bde5299a49448a60bc1121755a95]) successfully deleted
I20260928 10:56:09.137054 23715 tablet_replica.cc:330] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: stopping tablet replica
I20260928 10:56:09.137744 23715 raft_consensus.cc:2229] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:09.138325 23715 raft_consensus.cc:2258] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:09.140852 23715 ts_tablet_manager.cc:1905] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 10:56:09 dist-test-slave-8w8g krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:56:09.149991 23715 ts_tablet_manager.cc:1918] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:09.150261 23715 log.cc:1196] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/5cb92757584d41f484f30379610bc86f
I20260928 10:56:09.150977 23715 ts_tablet_manager.cc:1939] T 5cb92757584d41f484f30379610bc86f P fbc65d12dc824e87a910cd5511aa3943: Deleting consensus metadata
I20260928 10:56:09.154326 23201 catalog_manager.cc:4841] TS fbc65d12dc824e87a910cd5511aa3943 (127.5.227.131:39477): tablet 5cb92757584d41f484f30379610bc86f (table test-table [id=6a39bde5299a49448a60bc1121755a95]) successfully deleted
I20260928 10:56:09.156401 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:09.112049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.5.227.129:41033)
0928 10:56:09.112313 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:09.112323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:09.133345 (+ 21022us) server_negotiation.cc:407] Connection header received
0928 10:56:09.133909 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:09.133915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:09.134057 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:09.134151 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:09.137015 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.138050 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.139887 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.140318 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.140473 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:09.146151 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:09.146169 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:09.146185 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:09.146232 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:09.151214 (+ 4982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:09.153033 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:09.153043 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:09.153046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:09.153273 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:09.153976 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:09.153983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:09.153988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:09.154608 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:09.155234 (+ 626us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:09.155624 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 10:56:09.155946 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:56:09.157445 23399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:09.298857 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 23276
I20260928 10:56:09.320994 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 23410
Sep 28 10:56:09 dist-test-slave-8w8g krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:56:09.350809 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:09.322030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.5.227.130:41887)
0928 10:56:09.322241 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:09.322247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:09.322268 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:56:09.322488 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:09.322492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:09.322567 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:09.322679 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:09.324171 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.325957 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.328352 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.328975 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.329195 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:09.342043 (+ 12848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:09.342068 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:09.342088 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:09.342138 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:09.346284 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:09.347212 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:09.347221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:09.347223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:09.347396 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:09.348293 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:09.348298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:09.348301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:09.348906 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:09.349312 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:09.349752 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 10:56:09.350024 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:56:09.352113 23533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:09.497151 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 23410
I20260928 10:56:09.520406 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 23544
Sep 28 10:56:09 dist-test-slave-8w8g krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:56:09.542992 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:09.522117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.5.227.131:39477)
0928 10:56:09.522676 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:09.522681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:09.522700 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:56:09.522760 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:09.522764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:09.522841 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:09.522964 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:09.524742 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.525525 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.527113 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:09.527575 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:09.527711 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:09.534334 (+ 6623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:09.534352 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:09.534368 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:09.534418 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:09.538882 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:09.539895 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:09.539904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:09.539907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:09.540095 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:09.540989 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:09.540993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:09.540996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:09.541419 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:09.541675 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:09.542372 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 10:56:09.542685 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":276,"threads_started":1}
I20260928 10:56:09.543880 23667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:09.655575 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 23544
I20260928 10:56:09.675246 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 23183
I20260928 10:56:09.676710 23245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:09.800074 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 23183
2026-09-28T10:56:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5043 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T10:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:09Z Disabled control of system clock
I20260928 10:56:09.890053 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:38635
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:56:10.293141 23727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:10.293500 23727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:10.293605 23727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:10.302450 23727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:10.302588 23727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:10.302634 23727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:10.302685 23727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:10.302723 23727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:10.314316 23727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:38635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:10.315795 23727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:10.317772 23727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:10.333590 23732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:10.333755 23733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:10.334432 23735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:10.335144 23727 server_base.cc:1034] running on GCE node
I20260928 10:56:10.336361 23727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:10.338122 23727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:10.339393 23727 hybrid_clock.cc:648] HybridClock initialized: now 1790592970339349 us; error 44 us; skew 500 ppm
I20260928 10:56:10.340034 23727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:10.343122 23727 webserver.cc:460] Webserver started at http://127.5.227.190:43701/ using document root <none> and password file <none>
I20260928 10:56:10.344070 23727 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:10.344204 23727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:10.344549 23727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:10.348266 23727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "e879604942414951957bc167d2db9e76"
format_stamp: "Formatted at 2026-09-28 10:56:10 on dist-test-slave-8w8g"
server_key: "e85e1d3537351894992711ea79737a75"
server_key_iv: "c22fc80d66675e7173a137ce891339e6"
server_key_version: "encryptionkey@0"
I20260928 10:56:10.349279 23727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "e879604942414951957bc167d2db9e76"
format_stamp: "Formatted at 2026-09-28 10:56:10 on dist-test-slave-8w8g"
server_key: "e85e1d3537351894992711ea79737a75"
server_key_iv: "c22fc80d66675e7173a137ce891339e6"
server_key_version: "encryptionkey@0"
I20260928 10:56:10.355993 23727 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260928 10:56:10.360455 23741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:10.362519 23727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:56:10.362777 23727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "e879604942414951957bc167d2db9e76"
format_stamp: "Formatted at 2026-09-28 10:56:10 on dist-test-slave-8w8g"
server_key: "e85e1d3537351894992711ea79737a75"
server_key_iv: "c22fc80d66675e7173a137ce891339e6"
server_key_version: "encryptionkey@0"
I20260928 10:56:10.363099 23727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:10.391119 23727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:10.392334 23727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:10.392699 23727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:10.418076 23727 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:38635
I20260928 10:56:10.418081 23792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:38635 every 8 connection(s)
I20260928 10:56:10.419757 23727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:10.421942 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 23727
I20260928 10:56:10.422163 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:10.422857 6030 external_mini_cluster.cc:1428] Setting key c274371f1d1f32beb30d3bc05359505f
I20260928 10:56:10.426064 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:10.439059 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76: Bootstrap starting.
I20260928 10:56:10.439106 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:10.425307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.5.227.190:38635)
0928 10:56:10.426521 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:10.426534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:10.426599 (+ 65us) server_negotiation.cc:407] Connection header received
0928 10:56:10.428399 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:10.428423 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:10.428958 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:10.429414 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:10.430060 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:10.431806 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:10.433488 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:10.434637 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:10.434917 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:10.436519 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:10.436532 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:10.436549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:10.437188 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:10.437272 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:10.437296 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:10.437443 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:10.437691 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 10:56:10.438082 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":420,"threads_started":1}
I20260928 10:56:10.442832 23793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:10.444192 23793 log.cc:824] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:10.447857 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76: No bootstrap required, opened a new log
I20260928 10:56:10.457482 23793 raft_consensus.cc:348] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879604942414951957bc167d2db9e76" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38635 } }
I20260928 10:56:10.457926 23793 raft_consensus.cc:374] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:10.458006 23793 raft_consensus.cc:729] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e879604942414951957bc167d2db9e76, State: Initialized, Role: FOLLOWER
I20260928 10:56:10.458664 23793 consensus_queue.cc:260] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879604942414951957bc167d2db9e76" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38635 } }
I20260928 10:56:10.458917 23793 raft_consensus.cc:388] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:10.459036 23793 raft_consensus.cc:482] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:10.459189 23793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:10.462303 23793 raft_consensus.cc:504] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879604942414951957bc167d2db9e76" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38635 } }
I20260928 10:56:10.463080 23793 leader_election.cc:302] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e879604942414951957bc167d2db9e76; no voters:
I20260928 10:56:10.464036 23793 leader_election.cc:288] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:10.464232 23798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:10.465054 23798 raft_consensus.cc:686] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [term 1 LEADER]: Becoming Leader. State: Replica: e879604942414951957bc167d2db9e76, State: Running, Role: LEADER
I20260928 10:56:10.465763 23798 consensus_queue.cc:237] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879604942414951957bc167d2db9e76" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38635 } }
I20260928 10:56:10.466547 23793 sys_catalog.cc:564] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:10.470180 23800 sys_catalog.cc:455] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e879604942414951957bc167d2db9e76. Latest consensus state: current_term: 1 leader_uuid: "e879604942414951957bc167d2db9e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879604942414951957bc167d2db9e76" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38635 } } }
I20260928 10:56:10.470227 23799 sys_catalog.cc:455] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e879604942414951957bc167d2db9e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879604942414951957bc167d2db9e76" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 38635 } } }
I20260928 10:56:10.470834 23799 sys_catalog.cc:458] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:10.471637 23800 sys_catalog.cc:458] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:10.471962 23807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:10.478540 23807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:10.490427 23807 catalog_manager.cc:1347] Generated new cluster ID: 436537d7a988403880e25287aa339ddc
I20260928 10:56:10.490644 23807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:10.505445 23807 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:10.506997 23807 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:10.523284 23807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76: Generated new TSK 0
I20260928 10:56:10.524832 23807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:10.529743 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38635
--builtin_ntp_servers=127.5.227.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:56:10.913574 23817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:10.913903 23817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:10.914083 23817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:10.922538 23817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:10.922690 23817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:10.922878 23817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:56:10.933710 23817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:10.936128 23817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:10.938792 23817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:10.954860 23825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:10.954969 23823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:10.954947 23822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:10.956161 23817 server_base.cc:1034] running on GCE node
I20260928 10:56:10.957016 23817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:10.958212 23817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:10.959581 23817 hybrid_clock.cc:648] HybridClock initialized: now 1790592970959499 us; error 79 us; skew 500 ppm
I20260928 10:56:10.960186 23817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:10.964011 23817 webserver.cc:460] Webserver started at http://127.5.227.129:38931/ using document root <none> and password file <none>
I20260928 10:56:10.964874 23817 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:10.964982 23817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:10.965405 23817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:10.970038 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "6297d8dbaca54bab8bf686583c0c3fac"
format_stamp: "Formatted at 2026-09-28 10:56:10 on dist-test-slave-8w8g"
server_key: "69987aaebb74930d8f47ec59fedb2ef3"
server_key_iv: "1828dee7f38709b035d3be2a506cfa23"
server_key_version: "encryptionkey@0"
I20260928 10:56:10.971280 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "6297d8dbaca54bab8bf686583c0c3fac"
format_stamp: "Formatted at 2026-09-28 10:56:10 on dist-test-slave-8w8g"
server_key: "69987aaebb74930d8f47ec59fedb2ef3"
server_key_iv: "1828dee7f38709b035d3be2a506cfa23"
server_key_version: "encryptionkey@0"
I20260928 10:56:10.979221 23817 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260928 10:56:10.985123 23831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:10.987152 23817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:56:10.987416 23817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "6297d8dbaca54bab8bf686583c0c3fac"
format_stamp: "Formatted at 2026-09-28 10:56:10 on dist-test-slave-8w8g"
server_key: "69987aaebb74930d8f47ec59fedb2ef3"
server_key_iv: "1828dee7f38709b035d3be2a506cfa23"
server_key_version: "encryptionkey@0"
I20260928 10:56:10.987761 23817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:11.006896 23817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:11.008087 23817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:11.008479 23817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:11.009846 23817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:11.011842 23817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:11.011952 23817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:56:11.012040 23817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:11.012099 23817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:11.056414 23817 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:33515
I20260928 10:56:11.056468 23943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:33515 every 8 connection(s)
I20260928 10:56:11.057904 23817 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:56:11.059998 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 23817
I20260928 10:56:11.060191 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:56:11.060667 6030 external_mini_cluster.cc:1428] Setting key 43b25084915eb927a56dc673d4f104d9
I20260928 10:56:11.064344 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38635
--builtin_ntp_servers=127.5.227.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:56:11.075852 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:11.061448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:53803 (local address 127.5.227.190:38635)
0928 10:56:11.062683 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:11.062688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:11.063849 (+ 1161us) server_negotiation.cc:407] Connection header received
0928 10:56:11.066300 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:11.066306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:11.066433 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:11.066576 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:11.068183 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.068975 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.071842 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.072456 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.072673 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:11.074665 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:11.074687 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:11.074696 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:11.074726 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:11.074784 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:11.074806 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:11.074908 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:11.075232 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 10:56:11.075367 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":955,"threads_started":1}
I20260928 10:56:11.077005 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:11.062470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38635 (local address 127.5.227.129:53803)
0928 10:56:11.063539 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:11.063589 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:11.065933 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:11.066874 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:11.066885 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:11.067066 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:11.067938 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:11.067972 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.069194 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.069198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:11.071595 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:11.071615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.072801 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.072809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:11.073073 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:11.074210 (+ 1137us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:11.074213 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:11.074265 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:11.074373 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:11.074966 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:11.074979 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:11.074988 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 10:56:11.075184 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 10:56:11.075608 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":344,"threads_started":1}
I20260928 10:56:11.079015 23944 heartbeater.cc:347] Connected to a master server at 127.5.227.190:38635
I20260928 10:56:11.079375 23944 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:11.080332 23944 heartbeater.cc:511] Master 127.5.227.190:38635 requested a full tablet report, sending...
I20260928 10:56:11.083598 23758 ts_manager.cc:194] Registered new tserver with Master: 6297d8dbaca54bab8bf686583c0c3fac (127.5.227.129:33515)
I20260928 10:56:11.085884 23758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.129:53803
W20260928 10:56:11.458266 23948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:11.458607 23948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:11.458760 23948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:11.466383 23948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:11.466511 23948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:11.466632 23948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:56:11.477046 23948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:11.478446 23948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:11.480944 23948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:11.497661 23953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:11.497611 23956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:11.497534 23954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:11.499596 23948 server_base.cc:1034] running on GCE node
I20260928 10:56:11.500427 23948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:11.501806 23948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:11.503172 23948 hybrid_clock.cc:648] HybridClock initialized: now 1790592971503137 us; error 65 us; skew 500 ppm
I20260928 10:56:11.503563 23948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:11.506508 23948 webserver.cc:460] Webserver started at http://127.5.227.130:36399/ using document root <none> and password file <none>
I20260928 10:56:11.507242 23948 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:11.507347 23948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:11.507656 23948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:11.510794 23948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "4b9b4e4132bc4741bc452bc66aa0b194"
format_stamp: "Formatted at 2026-09-28 10:56:11 on dist-test-slave-8w8g"
server_key: "305372f38593e2d1478449570b70112e"
server_key_iv: "cf95685da810a957dbdb56028eae7265"
server_key_version: "encryptionkey@0"
I20260928 10:56:11.511678 23948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "4b9b4e4132bc4741bc452bc66aa0b194"
format_stamp: "Formatted at 2026-09-28 10:56:11 on dist-test-slave-8w8g"
server_key: "305372f38593e2d1478449570b70112e"
server_key_iv: "cf95685da810a957dbdb56028eae7265"
server_key_version: "encryptionkey@0"
I20260928 10:56:11.517701 23948 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 10:56:11.522354 23962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:11.524312 23948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:11.524530 23948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "4b9b4e4132bc4741bc452bc66aa0b194"
format_stamp: "Formatted at 2026-09-28 10:56:11 on dist-test-slave-8w8g"
server_key: "305372f38593e2d1478449570b70112e"
server_key_iv: "cf95685da810a957dbdb56028eae7265"
server_key_version: "encryptionkey@0"
I20260928 10:56:11.524765 23948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:11.539862 23948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:11.540767 23948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:11.541108 23948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:11.542241 23948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:11.544296 23948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:11.544405 23948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:11.544718 23948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:11.544828 23948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:11.591099 23948 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:38967
I20260928 10:56:11.591115 24074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:38967 every 8 connection(s)
I20260928 10:56:11.592625 23948 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:56:11.598471 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 23948
I20260928 10:56:11.598695 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:56:11.599341 6030 external_mini_cluster.cc:1428] Setting key 1a7958d9afb9c8fb6dae637d215a3b04
I20260928 10:56:11.602717 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38635
--builtin_ntp_servers=127.5.227.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:56:11.609553 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:11.595990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:48375 (local address 127.5.227.190:38635)
0928 10:56:11.596539 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:11.596544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:11.598093 (+ 1549us) server_negotiation.cc:407] Connection header received
0928 10:56:11.600911 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:11.600917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:11.601022 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:11.601171 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:11.602581 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.603520 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.605767 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.606257 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.606433 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:11.608486 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:11.608493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:11.608496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:11.608521 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:11.608580 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:11.608599 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:11.608716 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:11.608995 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 10:56:11.609122 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260928 10:56:11.610782 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:11.596694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38635 (local address 127.5.227.130:48375)
0928 10:56:11.597802 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:11.597851 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:11.600593 (+ 2742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:11.601511 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:11.601527 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:11.601698 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:11.602374 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:11.602395 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.603715 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.603720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:11.605437 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:11.605455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:11.606572 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:11.606581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:11.606865 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:11.607981 (+ 1116us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:11.607984 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:11.608083 (+ 99us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:11.608222 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:11.608717 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:11.608733 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:11.608745 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:56:11.608970 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 10:56:11.609470 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":334,"threads_started":1}
I20260928 10:56:11.612634 24075 heartbeater.cc:347] Connected to a master server at 127.5.227.190:38635
I20260928 10:56:11.613003 24075 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:11.614043 24075 heartbeater.cc:511] Master 127.5.227.190:38635 requested a full tablet report, sending...
I20260928 10:56:11.616088 23758 ts_manager.cc:194] Registered new tserver with Master: 4b9b4e4132bc4741bc452bc66aa0b194 (127.5.227.130:38967)
I20260928 10:56:11.617379 23758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.130:48375
W20260928 10:56:11.986663 24079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:11.986984 24079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:11.987078 24079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:11.995012 24079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:11.995137 24079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:11.995255 24079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:56:12.005915 24079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:12.007299 24079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:12.009166 24079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:12.026288 24084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:12.026247 24085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:12.026362 24087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:12.028017 24079 server_base.cc:1034] running on GCE node
I20260928 10:56:12.028837 24079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:12.030160 24079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:12.031415 24079 hybrid_clock.cc:648] HybridClock initialized: now 1790592972031324 us; error 88 us; skew 500 ppm
I20260928 10:56:12.031898 24079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:12.035310 24079 webserver.cc:460] Webserver started at http://127.5.227.131:36717/ using document root <none> and password file <none>
I20260928 10:56:12.036267 24079 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:12.036401 24079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:12.036798 24079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:12.040275 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "524a2d1574ec4e31af9bbca7d4697e17"
format_stamp: "Formatted at 2026-09-28 10:56:12 on dist-test-slave-8w8g"
server_key: "1788b432c6c75013445ed91cb4110bef"
server_key_iv: "260b1285536b3138a0edc0b97db820db"
server_key_version: "encryptionkey@0"
I20260928 10:56:12.041261 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "524a2d1574ec4e31af9bbca7d4697e17"
format_stamp: "Formatted at 2026-09-28 10:56:12 on dist-test-slave-8w8g"
server_key: "1788b432c6c75013445ed91cb4110bef"
server_key_iv: "260b1285536b3138a0edc0b97db820db"
server_key_version: "encryptionkey@0"
I20260928 10:56:12.048326 24079 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260928 10:56:12.052827 24093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:12.054571 24079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:56:12.054840 24079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "524a2d1574ec4e31af9bbca7d4697e17"
format_stamp: "Formatted at 2026-09-28 10:56:12 on dist-test-slave-8w8g"
server_key: "1788b432c6c75013445ed91cb4110bef"
server_key_iv: "260b1285536b3138a0edc0b97db820db"
server_key_version: "encryptionkey@0"
I20260928 10:56:12.055125 24079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:12.090245 23944 heartbeater.cc:503] Master 127.5.227.190:38635 was elected leader, sending a full tablet report...
I20260928 10:56:12.101881 24079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:12.102763 24079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:12.103072 24079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:12.104156 24079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:12.106343 24079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:12.106451 24079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:12.106534 24079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:12.106678 24079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:12.153095 24079 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:37177
I20260928 10:56:12.153108 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:37177 every 8 connection(s)
I20260928 10:56:12.154644 24079 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:56:12.156229 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 24079
I20260928 10:56:12.156405 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:56:12.156973 6030 external_mini_cluster.cc:1428] Setting key 3da29e18eced7a396e74f3369e3b21c5
I20260928 10:56:12.172930 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.158164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:59973 (local address 127.5.227.190:38635)
0928 10:56:12.159257 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.159263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.161020 (+ 1757us) server_negotiation.cc:407] Connection header received
0928 10:56:12.163773 (+ 2753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.163780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.163896 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.164038 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:12.165788 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.167299 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.169285 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.169958 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.170113 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.171892 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:12.171899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:12.171902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:12.171923 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:12.171965 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:12.171978 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:12.172103 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.172383 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.172499 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":868,"threads_started":1}
I20260928 10:56:12.174827 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.159324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:38635 (local address 127.5.227.131:59973)
0928 10:56:12.160826 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.160891 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:12.163292 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:12.164341 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:12.164351 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:12.164464 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:12.165297 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.165320 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.167499 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.167505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.169082 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.169100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.170233 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.170241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.170529 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.171440 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:12.171443 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:12.171491 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:12.171628 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:12.172106 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:12.172122 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:12.172132 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 10:56:12.172340 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 10:56:12.172865 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":483,"threads_started":1}
I20260928 10:56:12.176785 24206 heartbeater.cc:347] Connected to a master server at 127.5.227.190:38635
I20260928 10:56:12.177186 24206 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:12.178086 24206 heartbeater.cc:511] Master 127.5.227.190:38635 requested a full tablet report, sending...
I20260928 10:56:12.180003 23758 ts_manager.cc:194] Registered new tserver with Master: 524a2d1574ec4e31af9bbca7d4697e17 (127.5.227.131:37177)
I20260928 10:56:12.180918 23758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.131:59973
I20260928 10:56:12.188378 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:56:12.202411 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.194013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.5.227.190:38635)
0928 10:56:12.194349 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.194360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.194942 (+ 582us) server_negotiation.cc:407] Connection header received
0928 10:56:12.195391 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.195403 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.195503 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.195845 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:12.196760 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.197637 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.199539 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.200059 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.200232 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.201557 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:12.201563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:12.201567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:12.201590 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:12.201626 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:12.201640 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:12.201755 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.201928 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.202044 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 10:56:12.208127 23758 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:56:12.215469 23758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:56:12.264259 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.234161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46914)
0928 10:56:12.247095 (+ 12934us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.247138 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:12.247306 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:12.248997 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:12.249007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:12.249043 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:12.249576 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.249597 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.253845 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.253857 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.259761 (+ 5904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.259778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.263109 (+ 3331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.263118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.263344 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.263382 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:56:12.263455 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 10:56:12.263760 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12734,"mutex_wait_us":16,"thread_start_us":11392,"threads_started":1}
I20260928 10:56:12.264292 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.233366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:37177 (local address 127.0.0.1:48938)
0928 10:56:12.246242 (+ 12876us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.246320 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:12.246583 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:12.249488 (+ 2905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:12.249496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:12.249559 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:12.250111 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.250128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.255978 (+ 5850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.255988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.259362 (+ 3374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.259382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.262600 (+ 3218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.262613 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.263171 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.263304 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 10:56:12.263461 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 10:56:12.263766 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12418,"mutex_wait_us":5,"thread_start_us":13141,"threads_started":1}
I20260928 10:56:12.265573 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.235331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.5.227.129:33515)
0928 10:56:12.248278 (+ 12947us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.248294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.248333 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:56:12.248453 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.248462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.248734 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.248974 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:12.249894 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.253607 (+ 3713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.260249 (+ 6642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.262821 (+ 2572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.263785 (+ 964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.264227 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.264325 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.265123 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12656,"thread_start_us":12459,"threads_started":1}
I20260928 10:56:12.268705 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.233708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.5.227.131:37177)
0928 10:56:12.248764 (+ 15056us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.248777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.248821 (+ 44us) server_negotiation.cc:407] Connection header received
0928 10:56:12.248938 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.248948 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.249209 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.250364 (+ 1155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:12.250727 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.255717 (+ 4990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.259779 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.262236 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.265373 (+ 3137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.265932 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.266495 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.267075 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14830,"thread_start_us":14622,"threads_started":1}
I20260928 10:56:12.271281 23879 tablet_service.cc:1467] Processing CreateTablet for tablet 89a14c171a3c4ff19bcab9c7f11fd221 (DEFAULT_TABLE table=test-table [id=bedd00e6d5b84fd59aa1428f683e7e86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:12.273430 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.249480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:38967 (local address 127.0.0.1:56726)
0928 10:56:12.257039 (+ 7559us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.257066 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:12.257276 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:12.258936 (+ 1660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:12.258941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:12.258972 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:12.259627 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.259648 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.262217 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.262223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.269246 (+ 7023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.269271 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.272311 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.272319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.272559 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.272596 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 10:56:12.272719 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 10:56:12.272885 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7359,"mutex_wait_us":739,"thread_start_us":7928,"threads_started":1}
I20260928 10:56:12.273850 23879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a14c171a3c4ff19bcab9c7f11fd221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:12.274549 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.247417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.5.227.130:38967)
0928 10:56:12.255455 (+ 8038us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.255467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.257812 (+ 2345us) server_negotiation.cc:407] Connection header received
0928 10:56:12.257949 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.257959 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.258304 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.258539 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:12.259923 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.261987 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.269651 (+ 7664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.271298 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.271456 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.271926 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.273978 (+ 2052us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.274203 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7836,"thread_start_us":8811,"threads_started":1}
I20260928 10:56:12.273814 24141 tablet_service.cc:1467] Processing CreateTablet for tablet 89a14c171a3c4ff19bcab9c7f11fd221 (DEFAULT_TABLE table=test-table [id=bedd00e6d5b84fd59aa1428f683e7e86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:12.277987 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a14c171a3c4ff19bcab9c7f11fd221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:12.279289 24010 tablet_service.cc:1467] Processing CreateTablet for tablet 89a14c171a3c4ff19bcab9c7f11fd221 (DEFAULT_TABLE table=test-table [id=bedd00e6d5b84fd59aa1428f683e7e86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:12.281100 24010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a14c171a3c4ff19bcab9c7f11fd221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:12.292456 24225 tablet_bootstrap.cc:492] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Bootstrap starting.
I20260928 10:56:12.297423 24225 tablet_bootstrap.cc:654] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:12.299788 24225 log.cc:824] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:12.300448 24226 tablet_bootstrap.cc:492] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Bootstrap starting.
I20260928 10:56:12.305826 24226 tablet_bootstrap.cc:654] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:12.306058 24225 tablet_bootstrap.cc:492] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: No bootstrap required, opened a new log
I20260928 10:56:12.306382 24225 ts_tablet_manager.cc:1397] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260928 10:56:12.307596 24226 log.cc:824] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:12.310372 24229 tablet_bootstrap.cc:492] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Bootstrap starting.
I20260928 10:56:12.314294 24226 tablet_bootstrap.cc:492] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: No bootstrap required, opened a new log
I20260928 10:56:12.314872 24226 ts_tablet_manager.cc:1397] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260928 10:56:12.315603 24229 tablet_bootstrap.cc:654] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:12.318960 24229 log.cc:824] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:12.320045 24225 raft_consensus.cc:348] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.320514 24225 raft_consensus.cc:374] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:12.320663 24225 raft_consensus.cc:729] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6297d8dbaca54bab8bf686583c0c3fac, State: Initialized, Role: FOLLOWER
I20260928 10:56:12.321933 24225 consensus_queue.cc:260] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.322959 24229 tablet_bootstrap.cc:492] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: No bootstrap required, opened a new log
I20260928 10:56:12.323432 24229 ts_tablet_manager.cc:1397] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260928 10:56:12.325870 24225 ts_tablet_manager.cc:1428] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260928 10:56:12.328759 24226 raft_consensus.cc:348] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.329442 24226 raft_consensus.cc:374] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:12.329571 24226 raft_consensus.cc:729] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524a2d1574ec4e31af9bbca7d4697e17, State: Initialized, Role: FOLLOWER
I20260928 10:56:12.330307 24226 consensus_queue.cc:260] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.332130 24206 heartbeater.cc:503] Master 127.5.227.190:38635 was elected leader, sending a full tablet report...
I20260928 10:56:12.332957 24226 ts_tablet_manager.cc:1428] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Time spent starting tablet: real 0.018s user 0.015s sys 0.002s
I20260928 10:56:12.335500 24229 raft_consensus.cc:348] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.336490 24229 raft_consensus.cc:374] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:12.336663 24229 raft_consensus.cc:729] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9b4e4132bc4741bc452bc66aa0b194, State: Initialized, Role: FOLLOWER
I20260928 10:56:12.338223 24229 consensus_queue.cc:260] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.339977 24075 heartbeater.cc:503] Master 127.5.227.190:38635 was elected leader, sending a full tablet report...
I20260928 10:56:12.340842 24229 ts_tablet_manager.cc:1428] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260928 10:56:12.344980 24076 tablet.cc:2369] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:56:12.406920 24207 tablet.cc:2369] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:12.456413 24233 raft_consensus.cc:482] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:56:12.456833 24233 raft_consensus.cc:504] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.459013 24233 leader_election.cc:288] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 524a2d1574ec4e31af9bbca7d4697e17 (127.5.227.131:37177), 6297d8dbaca54bab8bf686583c0c3fac (127.5.227.129:33515)
I20260928 10:56:12.470281 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.459590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:37771 (local address 127.5.227.131:37177)
0928 10:56:12.459877 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.459887 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.460634 (+ 747us) server_negotiation.cc:407] Connection header received
0928 10:56:12.460774 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.460778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.460851 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.460966 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:12.462157 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.463401 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.467258 (+ 3857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.468719 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.468923 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.468999 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.469795 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.469951 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 10:56:12.470280 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.459593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:52235 (local address 127.5.227.129:33515)
0928 10:56:12.459925 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.459933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.460518 (+ 585us) server_negotiation.cc:407] Connection header received
0928 10:56:12.460743 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.460749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.460852 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.460968 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:12.462157 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.463404 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.467258 (+ 3854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.468719 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.468923 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.469002 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.469785 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.470013 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 10:56:12.470384 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.459603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:37177 (local address 127.5.227.130:37771)
0928 10:56:12.460482 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.460499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:12.460642 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:12.461072 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:12.461078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:12.461112 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:12.461891 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.461908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.463618 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.463625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.466912 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.466936 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.469114 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.469125 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.469483 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.469544 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 10:56:12.469666 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 10:56:12.469891 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":262,"thread_start_us":540,"threads_started":1}
I20260928 10:56:12.470461 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.459399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.5.227.130:52235)
0928 10:56:12.460351 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.460373 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:12.460575 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:12.461072 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:12.461078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:12.461167 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:12.461896 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.461908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.463618 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.463625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.466912 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:12.466936 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.469114 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.469125 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:12.469487 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.469527 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:56:12.469666 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 10:56:12.469891 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":438,"spinlock_wait_cycles":24960,"thread_start_us":455,"threads_started":1}
I20260928 10:56:12.471840 23899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a14c171a3c4ff19bcab9c7f11fd221" candidate_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6297d8dbaca54bab8bf686583c0c3fac" is_pre_election: true
I20260928 10:56:12.472198 24161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a14c171a3c4ff19bcab9c7f11fd221" candidate_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524a2d1574ec4e31af9bbca7d4697e17" is_pre_election: true
I20260928 10:56:12.472563 23899 raft_consensus.cc:2454] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9b4e4132bc4741bc452bc66aa0b194 in term 0.
I20260928 10:56:12.472771 24161 raft_consensus.cc:2454] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9b4e4132bc4741bc452bc66aa0b194 in term 0.
I20260928 10:56:12.473626 23964 leader_election.cc:302] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b9b4e4132bc4741bc452bc66aa0b194, 524a2d1574ec4e31af9bbca7d4697e17, 6297d8dbaca54bab8bf686583c0c3fac; no voters:
I20260928 10:56:12.474139 24233 raft_consensus.cc:2781] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:56:12.474311 24233 raft_consensus.cc:482] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:56:12.474432 24233 raft_consensus.cc:3037] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:12.477705 24233 raft_consensus.cc:504] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.478551 24233 leader_election.cc:288] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [CANDIDATE]: Term 1 election: Requested vote from peers 524a2d1574ec4e31af9bbca7d4697e17 (127.5.227.131:37177), 6297d8dbaca54bab8bf686583c0c3fac (127.5.227.129:33515)
I20260928 10:56:12.479207 24161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a14c171a3c4ff19bcab9c7f11fd221" candidate_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "524a2d1574ec4e31af9bbca7d4697e17"
I20260928 10:56:12.479266 23899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a14c171a3c4ff19bcab9c7f11fd221" candidate_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6297d8dbaca54bab8bf686583c0c3fac"
I20260928 10:56:12.479498 24161 raft_consensus.cc:3037] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:12.479498 23899 raft_consensus.cc:3037] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:12.483431 24161 raft_consensus.cc:2454] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9b4e4132bc4741bc452bc66aa0b194 in term 1.
I20260928 10:56:12.483431 23899 raft_consensus.cc:2454] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9b4e4132bc4741bc452bc66aa0b194 in term 1.
I20260928 10:56:12.484361 23966 leader_election.cc:302] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9b4e4132bc4741bc452bc66aa0b194, 6297d8dbaca54bab8bf686583c0c3fac; no voters:
I20260928 10:56:12.484828 24233 raft_consensus.cc:2781] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:12.485422 24233 raft_consensus.cc:686] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 1 LEADER]: Becoming Leader. State: Replica: 4b9b4e4132bc4741bc452bc66aa0b194, State: Running, Role: LEADER
I20260928 10:56:12.486174 24233 consensus_queue.cc:237] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } }
I20260928 10:56:12.492329 23758 catalog_manager.cc:5495] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b9b4e4132bc4741bc452bc66aa0b194 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b9b4e4132bc4741bc452bc66aa0b194" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 38967 } health_report { overall_health: HEALTHY } } }
I20260928 10:56:12.549984 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.543890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.5.227.130:38967)
0928 10:56:12.544184 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.544191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.544237 (+ 46us) server_negotiation.cc:407] Connection header received
0928 10:56:12.544518 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.544522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.544633 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.544816 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:56:12.545193 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.545892 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.547532 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.548101 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.548282 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.548830 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:56:12.549357 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:56:12.549538 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.549596 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.549696 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 10:56:12.561105 23945 tablet.cc:2369] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:12.561691 23899 raft_consensus.cc:1261] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 1 FOLLOWER]: Refusing update from remote peer 4b9b4e4132bc4741bc452bc66aa0b194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:12.561679 24161 raft_consensus.cc:1261] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 1 FOLLOWER]: Refusing update from remote peer 4b9b4e4132bc4741bc452bc66aa0b194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:12.563158 24236 consensus_queue.cc:1035] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6297d8dbaca54bab8bf686583c0c3fac" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:12.563555 24233 consensus_queue.cc:1035] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524a2d1574ec4e31af9bbca7d4697e17" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 37177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:12.577540 24239 mvcc.cc:205] Tried to move back new op lower bound from 7334268815595925504 to 7334268815308967936. Current Snapshot: MvccSnapshot[applied={T|T < 7334268815595925504}]
I20260928 10:56:12.595547 23758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:12.595887 23758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:12.600863 23758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e879604942414951957bc167d2db9e76: Sending DeleteTablet for 3 replicas of tablet 89a14c171a3c4ff19bcab9c7f11fd221
I20260928 10:56:12.602276 24141 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a14c171a3c4ff19bcab9c7f11fd221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:12 UTC) from {username='slave'} at 127.0.0.1:48938
I20260928 10:56:12.602895 23879 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a14c171a3c4ff19bcab9c7f11fd221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:12 UTC) from {username='slave'} at 127.0.0.1:46914
I20260928 10:56:12.604219 24010 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a14c171a3c4ff19bcab9c7f11fd221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:12 UTC) from {username='slave'} at 127.0.0.1:56726
I20260928 10:56:12.612524 24250 tablet_replica.cc:330] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: stopping tablet replica
I20260928 10:56:12.613193 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 23817
I20260928 10:56:12.613147 24250 raft_consensus.cc:2229] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:56:12.613976 24250 raft_consensus.cc:2258] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:12.617234 24250 ts_tablet_manager.cc:1905] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:12.636304 24249 tablet_replica.cc:330] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: stopping tablet replica
I20260928 10:56:12.636870 24249 raft_consensus.cc:2229] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:12.637537 24249 raft_consensus.cc:2258] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:12.640025 24248 tablet_replica.cc:330] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: stopping tablet replica
I20260928 10:56:12.641266 24248 raft_consensus.cc:2229] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:12.642052 24248 raft_consensus.cc:2258] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:12.643365 24250 ts_tablet_manager.cc:1918] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:12.643636 24250 log.cc:1196] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/89a14c171a3c4ff19bcab9c7f11fd221
I20260928 10:56:12.644321 24250 ts_tablet_manager.cc:1939] T 89a14c171a3c4ff19bcab9c7f11fd221 P 4b9b4e4132bc4741bc452bc66aa0b194: Deleting consensus metadata
I20260928 10:56:12.644519 24249 ts_tablet_manager.cc:1905] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:12.645255 24248 ts_tablet_manager.cc:1905] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:12.646787 23743 catalog_manager.cc:4841] TS 4b9b4e4132bc4741bc452bc66aa0b194 (127.5.227.130:38967): tablet 89a14c171a3c4ff19bcab9c7f11fd221 (table test-table [id=bedd00e6d5b84fd59aa1428f683e7e86]) successfully deleted
I20260928 10:56:12.650030 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.616532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.5.227.129:33515)
0928 10:56:12.616746 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.616753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.629333 (+ 12580us) server_negotiation.cc:407] Connection header received
0928 10:56:12.629924 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.629934 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.630094 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.630197 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:12.631074 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.632443 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.638260 (+ 5817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.639235 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.639379 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.647854 (+ 8475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:12.647870 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:12.647891 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:12.648812 (+ 921us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:12.648899 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:12.648930 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:12.649058 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.649371 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.649517 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 10:56:12.651378 23939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:12.654444 24248 ts_tablet_manager.cc:1918] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:12.655001 24248 log.cc:1196] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/89a14c171a3c4ff19bcab9c7f11fd221
I20260928 10:56:12.655725 24248 ts_tablet_manager.cc:1939] T 89a14c171a3c4ff19bcab9c7f11fd221 P 524a2d1574ec4e31af9bbca7d4697e17: Deleting consensus metadata
I20260928 10:56:12.657756 23743 catalog_manager.cc:4841] TS 524a2d1574ec4e31af9bbca7d4697e17 (127.5.227.131:37177): tablet 89a14c171a3c4ff19bcab9c7f11fd221 (table test-table [id=bedd00e6d5b84fd59aa1428f683e7e86]) successfully deleted
I20260928 10:56:12.757810 24249 ts_tablet_manager.cc:1918] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:12.758107 24249 log.cc:1196] T 89a14c171a3c4ff19bcab9c7f11fd221 P 6297d8dbaca54bab8bf686583c0c3fac: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/89a14c171a3c4ff19bcab9c7f11fd221
I20260928 10:56:12.758517 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 23817
I20260928 10:56:12.779241 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 23948
W20260928 10:56:12.779179 23745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.227.129:33515 (error 108)
I20260928 10:56:12.781603 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.780716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46938)
0928 10:56:12.781271 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.781403 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":330,"threads_started":1}
W20260928 10:56:12.782218 23745 catalog_manager.cc:4584] TS 6297d8dbaca54bab8bf686583c0c3fac (127.5.227.129:33515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89a14c171a3c4ff19bcab9c7f11fd221: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
I20260928 10:56:12.789686 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.780635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.5.227.130:38967)
0928 10:56:12.781299 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.781305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.781325 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:56:12.781395 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.781400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.781516 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.781621 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:12.782147 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.783426 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.785574 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.786354 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.786547 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.787717 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:12.787761 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:12.787791 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:12.788705 (+ 914us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:12.788778 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:12.788806 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:12.788934 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.789106 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.789228 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537}
I20260928 10:56:12.790566 24070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:12.836156 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.835455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46950)
0928 10:56:12.835821 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.835927 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 10:56:12.837713 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.837239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46956)
0928 10:56:12.837443 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.837521 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 10:56:12.898840 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.898227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46970)
0928 10:56:12.898528 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.898642 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 10:56:12.900144 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.899675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46980)
0928 10:56:12.899878 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.899964 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 10:56:12.917729 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 23948
I20260928 10:56:12.939450 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 24079
I20260928 10:56:12.948036 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.940535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.5.227.131:37177)
0928 10:56:12.940777 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:12.940782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:12.940803 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:56:12.941063 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:12.941068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:12.941179 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:12.941274 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:12.941842 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.942620 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.944287 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:12.944887 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:12.945012 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:12.946337 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:12.946346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:12.946366 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:12.947036 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:12.947096 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:12.947118 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:12.947215 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:12.947400 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 10:56:12.947634 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 10:56:12.948948 24201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:12.980975 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.980406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46992)
0928 10:56:12.980713 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.980813 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260928 10:56:12.982365 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:12.981740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:33515 (local address 127.0.0.1:46996)
0928 10:56:12.981969 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:12.982173 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 10:56:13.050741 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 24079
I20260928 10:56:13.068993 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 23727
I20260928 10:56:13.070353 23788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:13.187345 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 23727
2026-09-28T10:56:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3382 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:13 dist-test-slave-8w8g krb5kdc[24260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:13 dist-test-slave-8w8g krb5kdc[24260](info): set up 2 sockets
Sep 28 10:56:13 dist-test-slave-8w8g krb5kdc[24260](info): commencing operation
krb5kdc: starting...
W20260928 10:56:14.365221 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:14 dist-test-slave-8w8g krb5kdc[24260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:56:14.665500 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:32869
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:32869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:15.076738 24273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:15.077088 24273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:15.077284 24273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:15.085630 24273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:15.085915 24273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:15.086225 24273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:15.086323 24273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:15.086421 24273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:15.100942 24273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:32869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:15.102749 24273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:15.105067 24273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:15.123026 24278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:15.123101 24279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:15.123342 24273 server_base.cc:1034] running on GCE node
W20260928 10:56:15.123311 24281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:15.125088 24273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:15.126871 24273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:15.128228 24273 hybrid_clock.cc:648] HybridClock initialized: now 1790592975128149 us; error 66 us; skew 500 ppm
I20260928 10:56:15.128733 24273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:15 dist-test-slave-8w8g krb5kdc[24260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:15.133991 24273 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:56:15.137032 24273 webserver.cc:460] Webserver started at http://127.5.227.190:32985/ using document root <none> and password file <none>
I20260928 10:56:15.138262 24273 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:15.138706 24273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:15.139191 24273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:15.143949 24273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "a8937996c9e245b78a09e8fe0aeb9e29"
format_stamp: "Formatted at 2026-09-28 10:56:15 on dist-test-slave-8w8g"
server_key: "e09e54810bbb845e020baa2e3b5bc4aa"
server_key_iv: "7183ee19f5ea30af7697fdd2aec6017e"
server_key_version: "encryptionkey@0"
I20260928 10:56:15.145110 24273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "a8937996c9e245b78a09e8fe0aeb9e29"
format_stamp: "Formatted at 2026-09-28 10:56:15 on dist-test-slave-8w8g"
server_key: "e09e54810bbb845e020baa2e3b5bc4aa"
server_key_iv: "7183ee19f5ea30af7697fdd2aec6017e"
server_key_version: "encryptionkey@0"
I20260928 10:56:15.153103 24273 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260928 10:56:15.157919 24288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:15.160064 24273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260928 10:56:15.160264 24273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "a8937996c9e245b78a09e8fe0aeb9e29"
format_stamp: "Formatted at 2026-09-28 10:56:15 on dist-test-slave-8w8g"
server_key: "e09e54810bbb845e020baa2e3b5bc4aa"
server_key_iv: "7183ee19f5ea30af7697fdd2aec6017e"
server_key_version: "encryptionkey@0"
I20260928 10:56:15.160476 24273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:15.193372 24273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:15.199628 24273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:15.200035 24273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:15.227802 24339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:32869 every 8 connection(s)
I20260928 10:56:15.227890 24273 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:32869
I20260928 10:56:15.229909 24273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:15.235323 24340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:15.238605 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 24273
I20260928 10:56:15.238838 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:15.239478 6030 external_mini_cluster.cc:1428] Setting key cab47eab2191ae74282180041171ee80
I20260928 10:56:15.251935 24340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29: Bootstrap starting.
I20260928 10:56:15.256557 24340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:15.258106 24340 log.cc:824] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:56:15 dist-test-slave-8w8g krb5kdc[24260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:15.261682 24340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29: No bootstrap required, opened a new log
I20260928 10:56:15.269912 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:15.242057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.5.227.190:32869)
0928 10:56:15.243198 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:15.243212 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:15.243284 (+ 72us) server_negotiation.cc:407] Connection header received
0928 10:56:15.245656 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:15.245688 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:15.246186 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:15.246598 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:15.249406 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:15.250881 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:15.253111 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:15.253936 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:15.254108 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:15.260156 (+ 6048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:15.260168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:15.260186 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:15.260229 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:15.264427 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:15.265635 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:15.265643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:15.265651 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:15.265861 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:15.266314 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:15.266319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:15.266321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:15.266971 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:15.267340 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:15.267898 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 10:56:15.268450 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":634,"threads_started":1}
I20260928 10:56:15.272303 24340 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32869 } }
I20260928 10:56:15.272889 24340 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:15.273020 24340 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8937996c9e245b78a09e8fe0aeb9e29, State: Initialized, Role: FOLLOWER
I20260928 10:56:15.273896 24340 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32869 } }
I20260928 10:56:15.274362 24340 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:15.274520 24340 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:15.274792 24340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:15.277844 24340 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32869 } }
I20260928 10:56:15.279058 24340 leader_election.cc:302] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8937996c9e245b78a09e8fe0aeb9e29; no voters:
I20260928 10:56:15.280037 24340 leader_election.cc:288] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:15.280531 24345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:15.282425 24345 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [term 1 LEADER]: Becoming Leader. State: Replica: a8937996c9e245b78a09e8fe0aeb9e29, State: Running, Role: LEADER
I20260928 10:56:15.283327 24345 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32869 } }
I20260928 10:56:15.284786 24340 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:15.288362 24347 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8937996c9e245b78a09e8fe0aeb9e29. Latest consensus state: current_term: 1 leader_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32869 } } }
I20260928 10:56:15.289062 24347 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:15.288465 24346 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8937996c9e245b78a09e8fe0aeb9e29" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 32869 } } }
I20260928 10:56:15.290934 24346 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:15.292089 24352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:15.299275 24352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:15.311477 24352 catalog_manager.cc:1347] Generated new cluster ID: 24d29498787d463bbb5b35a54d23e32b
I20260928 10:56:15.311632 24352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:15.350556 24352 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:15.352160 24352 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:15.369666 24352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29: Generated new TSK 0
I20260928 10:56:15.370733 24352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:56:15.446282 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32869
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:15.842105 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:15.842499 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:15.842680 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:15.850502 24366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:15.850697 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:15.850890 24366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:56:15.861395 24366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:15.862886 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:15.864269 24366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:15.879570 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:15.879863 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:15.879839 24374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:15.881021 24366 server_base.cc:1034] running on GCE node
I20260928 10:56:15.881699 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:15.882727 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:15.883948 24366 hybrid_clock.cc:648] HybridClock initialized: now 1790592975883889 us; error 42 us; skew 500 ppm
I20260928 10:56:15.884382 24366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:15 dist-test-slave-8w8g krb5kdc[24260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:15.888392 24366 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:56:15.890048 24366 webserver.cc:460] Webserver started at http://127.5.227.129:35337/ using document root <none> and password file <none>
I20260928 10:56:15.890781 24366 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:15.890899 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:15.891181 24366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:15.894670 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "39712c4e2cd146c295e69aedb6e6bdee"
format_stamp: "Formatted at 2026-09-28 10:56:15 on dist-test-slave-8w8g"
server_key: "553a58c55d8169c18b1dd2fdd22f7048"
server_key_iv: "b18d8e7c18523f3f2a4688b010eee069"
server_key_version: "encryptionkey@0"
I20260928 10:56:15.895520 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "39712c4e2cd146c295e69aedb6e6bdee"
format_stamp: "Formatted at 2026-09-28 10:56:15 on dist-test-slave-8w8g"
server_key: "553a58c55d8169c18b1dd2fdd22f7048"
server_key_iv: "b18d8e7c18523f3f2a4688b010eee069"
server_key_version: "encryptionkey@0"
I20260928 10:56:15.901930 24366 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260928 10:56:15.906664 24381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:15.908674 24366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:56:15.908901 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "39712c4e2cd146c295e69aedb6e6bdee"
format_stamp: "Formatted at 2026-09-28 10:56:15 on dist-test-slave-8w8g"
server_key: "553a58c55d8169c18b1dd2fdd22f7048"
server_key_iv: "b18d8e7c18523f3f2a4688b010eee069"
server_key_version: "encryptionkey@0"
I20260928 10:56:15.909153 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:15.939079 24366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:15.944662 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:15.945007 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:15.946295 24366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:15.948062 24366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:15.948174 24366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:15.948329 24366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:15.948405 24366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:15.989984 24366 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:36693
I20260928 10:56:15.990036 24493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:36693 every 8 connection(s)
I20260928 10:56:15.991389 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:56:16.000144 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 24366
I20260928 10:56:16.000434 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:56:16.001109 6030 external_mini_cluster.cc:1428] Setting key 7f1072ef77ab43eba137f8d7f8055a62
Sep 28 10:56:16 dist-test-slave-8w8g krb5kdc[24260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:16.019771 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:15.994819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:59253 (local address 127.5.227.190:32869)
0928 10:56:15.995385 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:15.995390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:15.996753 (+ 1363us) server_negotiation.cc:407] Connection header received
0928 10:56:15.999039 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:15.999045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:15.999200 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:15.999338 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:16.001343 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.002097 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.005137 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.005914 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.006116 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:16.010366 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:16.010377 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:16.010384 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:16.010440 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:16.014349 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:16.015647 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:16.015657 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:16.015664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:16.015875 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:16.016431 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:16.016439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:16.016446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:16.016869 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:16.017254 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:16.018302 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0928 10:56:16.018751 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":335,"threads_started":1}
I20260928 10:56:16.020277 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:15.995481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:32869 (local address 127.5.227.129:59253)
0928 10:56:15.996311 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:15.996362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:15.998676 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:15.999635 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:15.999645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:16.000396 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:16.001107 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:16.001128 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.002383 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.002388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:16.004899 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:16.004923 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.006210 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.006219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:16.006579 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:16.007515 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:16.007550 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:16.010018 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:16.014762 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:16.014773 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:16.014811 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:16.015454 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:16.016031 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:16.016036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:16.016038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:16.016280 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:16.017300 (+ 1020us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:16.017308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:16.017836 (+ 528us) client_negotiation.cc:769] Sending connection context
0928 10:56:16.018127 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 10:56:16.018597 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":352,"threads_started":1}
I20260928 10:56:16.022456 24494 heartbeater.cc:347] Connected to a master server at 127.5.227.190:32869
I20260928 10:56:16.023062 24494 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:16.023970 24494 heartbeater.cc:511] Master 127.5.227.190:32869 requested a full tablet report, sending...
I20260928 10:56:16.026688 24305 ts_manager.cc:194] Registered new tserver with Master: 39712c4e2cd146c295e69aedb6e6bdee (127.5.227.129:36693)
I20260928 10:56:16.028394 24305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:59253
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:56:16.080533 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32869
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:16.455148 24500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:16.455456 24500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:16.455616 24500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:16.463145 24500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:16.463277 24500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:16.463480 24500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:56:16.473785 24500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:16.475255 24500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:16.476713 24500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:16.493513 24508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:16.493716 24506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:16.494050 24505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:16.495488 24500 server_base.cc:1034] running on GCE node
I20260928 10:56:16.496347 24500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:16.497720 24500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:16.499085 24500 hybrid_clock.cc:648] HybridClock initialized: now 1790592976498976 us; error 45 us; skew 500 ppm
I20260928 10:56:16.499511 24500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:16 dist-test-slave-8w8g krb5kdc[24260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592976, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:16.503369 24500 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:56:16.505493 24500 webserver.cc:460] Webserver started at http://127.5.227.130:41863/ using document root <none> and password file <none>
I20260928 10:56:16.506382 24500 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:16.506568 24500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:16.506955 24500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:16.512147 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "0f92968f0a444f0da79a092ca1197ba5"
format_stamp: "Formatted at 2026-09-28 10:56:16 on dist-test-slave-8w8g"
server_key: "39aa164bfa65864a633e253089b7db33"
server_key_iv: "bee3bac53bc1aeb70de9ae2271da00f7"
server_key_version: "encryptionkey@0"
I20260928 10:56:16.513772 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "0f92968f0a444f0da79a092ca1197ba5"
format_stamp: "Formatted at 2026-09-28 10:56:16 on dist-test-slave-8w8g"
server_key: "39aa164bfa65864a633e253089b7db33"
server_key_iv: "bee3bac53bc1aeb70de9ae2271da00f7"
server_key_version: "encryptionkey@0"
I20260928 10:56:16.520313 24500 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:56:16.524616 24515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:16.526362 24500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:56:16.526600 24500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "0f92968f0a444f0da79a092ca1197ba5"
format_stamp: "Formatted at 2026-09-28 10:56:16 on dist-test-slave-8w8g"
server_key: "39aa164bfa65864a633e253089b7db33"
server_key_iv: "bee3bac53bc1aeb70de9ae2271da00f7"
server_key_version: "encryptionkey@0"
I20260928 10:56:16.526865 24500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:16.547508 24500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:16.552932 24500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:16.553299 24500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:16.554483 24500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:16.556313 24500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:16.556432 24500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:16.556609 24500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:16.556691 24500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:16.600083 24500 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:41497
I20260928 10:56:16.600154 24627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:41497 every 8 connection(s)
I20260928 10:56:16.601722 24500 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:56:16.612288 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 24500
I20260928 10:56:16.612521 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:56:16.613103 6030 external_mini_cluster.cc:1428] Setting key 13803c61d04fac6049140f1aa39df119
Sep 28 10:56:16 dist-test-slave-8w8g krb5kdc[24260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592976, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:16.628501 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:16.604991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:55607 (local address 127.5.227.190:32869)
0928 10:56:16.605478 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:16.605483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:16.606389 (+ 906us) server_negotiation.cc:407] Connection header received
0928 10:56:16.608484 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:16.608491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:16.608586 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:16.608978 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:16.610951 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.611706 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.613663 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.614336 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.614481 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:16.620730 (+ 6249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:16.620738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:16.620742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:16.620785 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:16.624812 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:16.625925 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:16.625932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:16.625935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:16.626086 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:16.626559 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:16.626565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:16.626570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:16.626876 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:16.627048 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:16.627863 (+ 815us) server_negotiation.cc:299] Negotiation successful
0928 10:56:16.628119 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260928 10:56:16.629583 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:16.605426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:32869 (local address 127.5.227.130:55607)
0928 10:56:16.606212 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:16.606252 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:16.608131 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:16.609267 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:16.609277 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:16.610065 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:16.610710 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:16.610732 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.611959 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.611963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:16.613492 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:16.613507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:16.614942 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:16.615010 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:16.615660 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:16.617302 (+ 1642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:16.617336 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:16.620257 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:16.625040 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:16.625052 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:16.625080 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:16.625588 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:16.626213 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:16.626218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:16.626222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:16.626436 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:16.627059 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:16.627070 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:16.627457 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 10:56:16.627771 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 10:56:16.628376 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":323,"threads_started":1}
I20260928 10:56:16.631378 24628 heartbeater.cc:347] Connected to a master server at 127.5.227.190:32869
I20260928 10:56:16.631929 24628 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:16.632867 24628 heartbeater.cc:511] Master 127.5.227.190:32869 requested a full tablet report, sending...
I20260928 10:56:16.634701 24305 ts_manager.cc:194] Registered new tserver with Master: 0f92968f0a444f0da79a092ca1197ba5 (127.5.227.130:41497)
I20260928 10:56:16.636255 24305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:55607
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:56:16.697402 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32869
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:56:17.032387 24494 heartbeater.cc:503] Master 127.5.227.190:32869 was elected leader, sending a full tablet report...
W20260928 10:56:17.097659 24634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:17.097981 24634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:17.098079 24634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:17.106904 24634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:17.107048 24634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:17.107167 24634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:56:17.117955 24634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:32869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:17.119426 24634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:17.120795 24634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:17.136468 24640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:17.136696 24642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:17.136655 24639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:17.137447 24634 server_base.cc:1034] running on GCE node
I20260928 10:56:17.138168 24634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:17.139304 24634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:17.140713 24634 hybrid_clock.cc:648] HybridClock initialized: now 1790592977140620 us; error 97 us; skew 500 ppm
I20260928 10:56:17.141294 24634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:17 dist-test-slave-8w8g krb5kdc[24260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592977, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:17.146382 24634 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:56:17.148689 24634 webserver.cc:460] Webserver started at http://127.5.227.131:45837/ using document root <none> and password file <none>
I20260928 10:56:17.149499 24634 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:17.149819 24634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:17.150308 24634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:17.153847 24634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "959255e83bc5449486114af5e0d5773e"
format_stamp: "Formatted at 2026-09-28 10:56:17 on dist-test-slave-8w8g"
server_key: "d44f2d663cc913f3fce5eaf70414536b"
server_key_iv: "fa4b34420b0246c40e2cccdcbf412ae9"
server_key_version: "encryptionkey@0"
I20260928 10:56:17.154896 24634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "959255e83bc5449486114af5e0d5773e"
format_stamp: "Formatted at 2026-09-28 10:56:17 on dist-test-slave-8w8g"
server_key: "d44f2d663cc913f3fce5eaf70414536b"
server_key_iv: "fa4b34420b0246c40e2cccdcbf412ae9"
server_key_version: "encryptionkey@0"
I20260928 10:56:17.161263 24634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 10:56:17.166657 24649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:17.168838 24634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:17.169107 24634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "959255e83bc5449486114af5e0d5773e"
format_stamp: "Formatted at 2026-09-28 10:56:17 on dist-test-slave-8w8g"
server_key: "d44f2d663cc913f3fce5eaf70414536b"
server_key_iv: "fa4b34420b0246c40e2cccdcbf412ae9"
server_key_version: "encryptionkey@0"
I20260928 10:56:17.169368 24634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:17.203234 24634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:17.208516 24634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:17.208861 24634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:17.210460 24634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:17.212110 24634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:17.212224 24634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:17.212383 24634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:17.212456 24634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:17.252439 24634 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:41971
I20260928 10:56:17.252492 24761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:41971 every 8 connection(s)
I20260928 10:56:17.253918 24634 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:56:17.262276 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 24634
I20260928 10:56:17.262492 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:56:17.263228 6030 external_mini_cluster.cc:1428] Setting key fe65074c16e339d9d6cfc0dd2e3e7941
Sep 28 10:56:17 dist-test-slave-8w8g krb5kdc[24260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592977, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:17.279441 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.257382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:49857 (local address 127.5.227.190:32869)
0928 10:56:17.257916 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.257922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.258910 (+ 988us) server_negotiation.cc:407] Connection header received
0928 10:56:17.261191 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.261199 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.261323 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.261469 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:17.264289 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.265382 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.267307 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.267827 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.267938 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.272296 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:17.272302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:17.272305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:17.272344 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:17.275385 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.276556 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.276561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.276564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.276803 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.277430 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.277434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.277436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.277745 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:17.277880 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.278787 (+ 907us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.279080 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":275,"threads_started":1}
I20260928 10:56:17.280539 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.257996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:32869 (local address 127.5.227.131:49857)
0928 10:56:17.258723 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:17.258765 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:17.260837 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:17.262056 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:17.262085 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:17.263055 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:17.264003 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.264059 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.265579 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.265585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.267154 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.267169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.268092 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.268099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.268334 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.269272 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:17.269302 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:17.271991 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:17.275585 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:17.275716 (+ 131us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:17.275798 (+ 82us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:17.276351 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:17.276949 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:17.276953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:17.276955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:17.277296 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:17.277879 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:17.277888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:17.278393 (+ 505us) client_negotiation.cc:769] Sending connection context
0928 10:56:17.278646 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 10:56:17.279226 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":278,"threads_started":1}
I20260928 10:56:17.282155 24762 heartbeater.cc:347] Connected to a master server at 127.5.227.190:32869
I20260928 10:56:17.282662 24762 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:17.283591 24762 heartbeater.cc:511] Master 127.5.227.190:32869 requested a full tablet report, sending...
I20260928 10:56:17.285553 24305 ts_manager.cc:194] Registered new tserver with Master: 959255e83bc5449486114af5e0d5773e (127.5.227.131:41971)
I20260928 10:56:17.286446 24305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:49857
I20260928 10:56:17.293098 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:56:17.314075 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.297970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.5.227.190:32869)
0928 10:56:17.298252 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.298260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.298587 (+ 327us) server_negotiation.cc:407] Connection header received
0928 10:56:17.298810 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.298815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.298945 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.299067 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:17.300661 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.301421 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.302825 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.303262 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.303432 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.306740 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:17.306746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:17.306750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:17.306797 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:17.310401 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.311472 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.311478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.311481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.311625 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.312174 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.312187 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.312189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.312536 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:17.312843 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.313363 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.313700 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 10:56:17.319111 24305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:56:17.326560 24305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:56:17.357501 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.341762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:41497 (local address 127.0.0.1:42324)
0928 10:56:17.343156 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:17.343199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:17.343445 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:17.344810 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:17.344818 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:17.344852 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:17.345374 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.345395 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.348384 (+ 2989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.348393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.352110 (+ 3717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.352141 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.355828 (+ 3687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.355839 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.356179 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.356262 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 10:56:17.356452 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 10:56:17.356801 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":4,"thread_start_us":1016,"threads_started":1}
I20260928 10:56:17.359179 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.341744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.5.227.130:41497)
0928 10:56:17.342587 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.342791 (+ 204us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.343323 (+ 532us) server_negotiation.cc:407] Connection header received
0928 10:56:17.343950 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.343962 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.344249 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.344494 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:17.345718 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.348098 (+ 2380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.352448 (+ 4350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.355061 (+ 2613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.355274 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.355609 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.358319 (+ 2710us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.358639 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":508,"threads_started":1}
I20260928 10:56:17.362162 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.343490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36693 (local address 127.0.0.1:35296)
0928 10:56:17.344140 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:17.344160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:17.344273 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:17.347864 (+ 3591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:17.347873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:17.347916 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:17.348618 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.348638 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.353699 (+ 5061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.353710 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.359152 (+ 5442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.359177 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.361394 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.361404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.361563 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.361590 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 10:56:17.361648 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 10:56:17.361815 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":342,"threads_started":1}
I20260928 10:56:17.363696 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.344131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.5.227.129:36693)
0928 10:56:17.346924 (+ 2793us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.346937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.346977 (+ 40us) server_negotiation.cc:407] Connection header received
0928 10:56:17.347104 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.347117 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.347509 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.349004 (+ 1495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:17.351233 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.353437 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.359481 (+ 6044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.361106 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.362926 (+ 1820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.363197 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.363276 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.363418 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2518,"thread_start_us":519,"threads_started":1}
I20260928 10:56:17.366006 24563 tablet_service.cc:1467] Processing CreateTablet for tablet fff6d38ff665411ab791de032d758aac (DEFAULT_TABLE table=test-table [id=314dd33f860c4f8b9af4f3cf0a1e7aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:17.367415 24563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6d38ff665411ab791de032d758aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:17.367849 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.345004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:41971 (local address 127.0.0.1:46116)
0928 10:56:17.357876 (+ 12872us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:17.357912 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:17.358065 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:17.360661 (+ 2596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:17.360671 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:17.360718 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:17.360987 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.361003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.363079 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.363088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.365008 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.365026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.366949 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.366956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.367155 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.367202 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 10:56:17.367309 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 10:56:17.367508 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12713,"thread_start_us":16206,"threads_started":1}
I20260928 10:56:17.368233 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.344715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.5.227.131:41971)
0928 10:56:17.359919 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.359928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.359977 (+ 49us) server_negotiation.cc:407] Connection header received
0928 10:56:17.360088 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.360096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.360385 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.360607 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:17.361235 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.362863 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.365183 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.366752 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.366894 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.367118 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.367385 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.367659 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14971,"thread_start_us":16101,"threads_started":1}
I20260928 10:56:17.367468 24429 tablet_service.cc:1467] Processing CreateTablet for tablet fff6d38ff665411ab791de032d758aac (DEFAULT_TABLE table=test-table [id=314dd33f860c4f8b9af4f3cf0a1e7aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:17.369611 24429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6d38ff665411ab791de032d758aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:17.372758 24697 tablet_service.cc:1467] Processing CreateTablet for tablet fff6d38ff665411ab791de032d758aac (DEFAULT_TABLE table=test-table [id=314dd33f860c4f8b9af4f3cf0a1e7aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:17.374408 24697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6d38ff665411ab791de032d758aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:17.380264 24781 tablet_bootstrap.cc:492] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Bootstrap starting.
I20260928 10:56:17.383678 24781 tablet_bootstrap.cc:654] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:17.384904 24781 log.cc:824] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:17.387216 24782 tablet_bootstrap.cc:492] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Bootstrap starting.
I20260928 10:56:17.390611 24781 tablet_bootstrap.cc:492] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: No bootstrap required, opened a new log
I20260928 10:56:17.390941 24781 ts_tablet_manager.cc:1397] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 10:56:17.392369 24782 tablet_bootstrap.cc:654] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:17.392424 24784 tablet_bootstrap.cc:492] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Bootstrap starting.
I20260928 10:56:17.393894 24782 log.cc:824] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:17.396732 24784 tablet_bootstrap.cc:654] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:17.397825 24782 tablet_bootstrap.cc:492] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: No bootstrap required, opened a new log
I20260928 10:56:17.398156 24782 ts_tablet_manager.cc:1397] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 10:56:17.398486 24784 log.cc:824] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:17.403084 24784 tablet_bootstrap.cc:492] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: No bootstrap required, opened a new log
I20260928 10:56:17.403534 24784 ts_tablet_manager.cc:1397] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260928 10:56:17.403895 24781 raft_consensus.cc:348] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.404422 24781 raft_consensus.cc:374] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:17.404523 24781 raft_consensus.cc:729] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f92968f0a444f0da79a092ca1197ba5, State: Initialized, Role: FOLLOWER
I20260928 10:56:17.405400 24781 consensus_queue.cc:260] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.406665 24628 heartbeater.cc:503] Master 127.5.227.190:32869 was elected leader, sending a full tablet report...
I20260928 10:56:17.407414 24781 ts_tablet_manager.cc:1428] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Time spent starting tablet: real 0.016s user 0.018s sys 0.000s
I20260928 10:56:17.409775 24782 raft_consensus.cc:348] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.410241 24782 raft_consensus.cc:374] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:17.410390 24782 raft_consensus.cc:729] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39712c4e2cd146c295e69aedb6e6bdee, State: Initialized, Role: FOLLOWER
I20260928 10:56:17.411185 24782 consensus_queue.cc:260] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.413318 24782 ts_tablet_manager.cc:1428] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260928 10:56:17.416986 24784 raft_consensus.cc:348] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.417583 24784 raft_consensus.cc:374] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:17.417726 24784 raft_consensus.cc:729] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959255e83bc5449486114af5e0d5773e, State: Initialized, Role: FOLLOWER
I20260928 10:56:17.418354 24784 consensus_queue.cc:260] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.419559 24762 heartbeater.cc:503] Master 127.5.227.190:32869 was elected leader, sending a full tablet report...
I20260928 10:56:17.420161 24784 ts_tablet_manager.cc:1428] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260928 10:56:17.495012 24495 tablet.cc:2369] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:56:17.505772 24763 tablet.cc:2369] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:17.538033 24788 raft_consensus.cc:482] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:56:17.538324 24788 raft_consensus.cc:504] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.540212 24788 leader_election.cc:288] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f92968f0a444f0da79a092ca1197ba5 (127.5.227.130:41497), 959255e83bc5449486114af5e0d5773e (127.5.227.131:41971)
I20260928 10:56:17.549538 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.540799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:60221 (local address 127.5.227.130:41497)
0928 10:56:17.541072 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.541081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.541676 (+ 595us) server_negotiation.cc:407] Connection header received
0928 10:56:17.542000 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.542006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.542145 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.542215 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:17.543092 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.544168 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.546893 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.548154 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.548323 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.548384 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.549189 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.549328 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:56:17.549817 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.540613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:41497 (local address 127.5.227.129:60221)
0928 10:56:17.541535 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:17.541560 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:17.541818 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:17.542435 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:17.542440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:17.542467 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:17.542925 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.542935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.544366 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.544372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.546697 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.546712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.548426 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.548434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.548932 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.548975 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:56:17.549129 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 10:56:17.549311 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":32,"thread_start_us":611,"threads_started":1}
I20260928 10:56:17.551342 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.542029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:60305 (local address 127.5.227.131:41971)
0928 10:56:17.542335 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.542343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.543177 (+ 834us) server_negotiation.cc:407] Connection header received
0928 10:56:17.543398 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.543403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.543500 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.543593 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:17.544367 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.545657 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.548342 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.550111 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.550350 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.550443 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.550883 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.551052 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 10:56:17.551636 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.541758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:41971 (local address 127.5.227.129:60305)
0928 10:56:17.543040 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:17.543059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:17.543222 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:17.543638 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:17.543643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:17.543669 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:17.544059 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.544071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.545858 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.545863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.548103 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:17.548122 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.550432 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.550438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:17.550643 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.550671 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 10:56:17.550766 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 10:56:17.551069 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":5,"thread_start_us":861,"threads_started":1}
I20260928 10:56:17.551626 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6d38ff665411ab791de032d758aac" candidate_uuid: "39712c4e2cd146c295e69aedb6e6bdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f92968f0a444f0da79a092ca1197ba5" is_pre_election: true
I20260928 10:56:17.552452 24583 raft_consensus.cc:2454] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39712c4e2cd146c295e69aedb6e6bdee in term 0.
I20260928 10:56:17.553295 24717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6d38ff665411ab791de032d758aac" candidate_uuid: "39712c4e2cd146c295e69aedb6e6bdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959255e83bc5449486114af5e0d5773e" is_pre_election: true
I20260928 10:56:17.553412 24384 leader_election.cc:302] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f92968f0a444f0da79a092ca1197ba5, 39712c4e2cd146c295e69aedb6e6bdee; no voters:
I20260928 10:56:17.554113 24717 raft_consensus.cc:2454] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39712c4e2cd146c295e69aedb6e6bdee in term 0.
I20260928 10:56:17.554167 24788 raft_consensus.cc:2781] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:56:17.554338 24788 raft_consensus.cc:482] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:56:17.554440 24788 raft_consensus.cc:3037] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:17.557056 24788 raft_consensus.cc:504] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.557788 24788 leader_election.cc:288] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [CANDIDATE]: Term 1 election: Requested vote from peers 0f92968f0a444f0da79a092ca1197ba5 (127.5.227.130:41497), 959255e83bc5449486114af5e0d5773e (127.5.227.131:41971)
I20260928 10:56:17.558465 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6d38ff665411ab791de032d758aac" candidate_uuid: "39712c4e2cd146c295e69aedb6e6bdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f92968f0a444f0da79a092ca1197ba5"
I20260928 10:56:17.558472 24717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6d38ff665411ab791de032d758aac" candidate_uuid: "39712c4e2cd146c295e69aedb6e6bdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959255e83bc5449486114af5e0d5773e"
I20260928 10:56:17.558729 24583 raft_consensus.cc:3037] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:17.558740 24717 raft_consensus.cc:3037] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:17.561271 24583 raft_consensus.cc:2454] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39712c4e2cd146c295e69aedb6e6bdee in term 1.
I20260928 10:56:17.561277 24717 raft_consensus.cc:2454] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39712c4e2cd146c295e69aedb6e6bdee in term 1.
I20260928 10:56:17.561842 24384 leader_election.cc:302] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f92968f0a444f0da79a092ca1197ba5, 39712c4e2cd146c295e69aedb6e6bdee; no voters:
I20260928 10:56:17.562201 24788 raft_consensus.cc:2781] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:17.562841 24788 raft_consensus.cc:686] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 1 LEADER]: Becoming Leader. State: Replica: 39712c4e2cd146c295e69aedb6e6bdee, State: Running, Role: LEADER
I20260928 10:56:17.563436 24788 consensus_queue.cc:237] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } }
I20260928 10:56:17.568733 24305 catalog_manager.cc:5495] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee reported cstate change: term changed from 0 to 1, leader changed from <none> to 39712c4e2cd146c295e69aedb6e6bdee (127.5.227.129). New cstate: current_term: 1 leader_uuid: "39712c4e2cd146c295e69aedb6e6bdee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39712c4e2cd146c295e69aedb6e6bdee" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 } health_report { overall_health: UNKNOWN } } }
W20260928 10:56:17.604483 24629 tablet.cc:2369] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:17.605314 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.597689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.5.227.129:36693)
0928 10:56:17.597890 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.597894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.597914 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:56:17.598010 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.598014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.598113 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.598248 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:56:17.598888 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.600192 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.601695 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.602403 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.602542 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.603315 (+ 773us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:56:17.604274 (+ 959us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:56:17.604496 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.604949 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.605018 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 10:56:17.615343 24717 raft_consensus.cc:1261] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 1 FOLLOWER]: Refusing update from remote peer 39712c4e2cd146c295e69aedb6e6bdee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:17.615658 24583 raft_consensus.cc:1261] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 1 FOLLOWER]: Refusing update from remote peer 39712c4e2cd146c295e69aedb6e6bdee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:17.616813 24792 consensus_queue.cc:1035] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [LEADER]: Connected to new peer: Peer: permanent_uuid: "959255e83bc5449486114af5e0d5773e" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:17.617244 24788 consensus_queue.cc:1035] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f92968f0a444f0da79a092ca1197ba5" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:17.630950 24795 mvcc.cc:205] Tried to move back new op lower bound from 7334268836298919936 to 7334268836104654848. Current Snapshot: MvccSnapshot[applied={T|T < 7334268836298919936}]
I20260928 10:56:17.647136 24304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:17.647485 24304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:17.651917 24304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8937996c9e245b78a09e8fe0aeb9e29: Sending DeleteTablet for 3 replicas of tablet fff6d38ff665411ab791de032d758aac
I20260928 10:56:17.653441 24563 tablet_service.cc:1514] Processing DeleteTablet for tablet fff6d38ff665411ab791de032d758aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:17 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:42324
I20260928 10:56:17.653786 24697 tablet_service.cc:1514] Processing DeleteTablet for tablet fff6d38ff665411ab791de032d758aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:17 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:46116
I20260928 10:56:17.653874 24429 tablet_service.cc:1514] Processing DeleteTablet for tablet fff6d38ff665411ab791de032d758aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:17 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:35296
I20260928 10:56:17.655853 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 24366
I20260928 10:56:17.665993 24805 tablet_replica.cc:330] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: stopping tablet replica
I20260928 10:56:17.666710 24805 raft_consensus.cc:2229] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:17.667446 24805 raft_consensus.cc:2258] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:17.669962 24805 ts_tablet_manager.cc:1905] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:17.679977 24806 tablet_replica.cc:330] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: stopping tablet replica
I20260928 10:56:17.680650 24806 raft_consensus.cc:2229] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:17.681389 24806 raft_consensus.cc:2258] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:17.685392 24805 ts_tablet_manager.cc:1918] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:17.686411 24807 tablet_replica.cc:330] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: stopping tablet replica
I20260928 10:56:17.687005 24807 raft_consensus.cc:2229] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:56:17.687955 24807 raft_consensus.cc:2258] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:17.688444 24805 log.cc:1196] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/fff6d38ff665411ab791de032d758aac
I20260928 10:56:17.688038 24806 ts_tablet_manager.cc:1905] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:17.689127 24805 ts_tablet_manager.cc:1939] T fff6d38ff665411ab791de032d758aac P 0f92968f0a444f0da79a092ca1197ba5: Deleting consensus metadata
I20260928 10:56:17.691425 24291 catalog_manager.cc:4841] TS 0f92968f0a444f0da79a092ca1197ba5 (127.5.227.130:41497): tablet fff6d38ff665411ab791de032d758aac (table test-table [id=314dd33f860c4f8b9af4f3cf0a1e7aac]) successfully deleted
I20260928 10:56:17.693248 24807 ts_tablet_manager.cc:1905] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:17.694187 24806 ts_tablet_manager.cc:1918] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:17.694416 24806 log.cc:1196] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/fff6d38ff665411ab791de032d758aac
I20260928 10:56:17.695266 24806 ts_tablet_manager.cc:1939] T fff6d38ff665411ab791de032d758aac P 959255e83bc5449486114af5e0d5773e: Deleting consensus metadata
I20260928 10:56:17.698133 24291 catalog_manager.cc:4841] TS 959255e83bc5449486114af5e0d5773e (127.5.227.131:41971): tablet fff6d38ff665411ab791de032d758aac (table test-table [id=314dd33f860c4f8b9af4f3cf0a1e7aac]) successfully deleted
Sep 28 10:56:17 dist-test-slave-8w8g krb5kdc[24260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:56:17.702927 24807 ts_tablet_manager.cc:1918] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:17.703373 24807 log.cc:1196] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/fff6d38ff665411ab791de032d758aac
I20260928 10:56:17.704332 24807 ts_tablet_manager.cc:1939] T fff6d38ff665411ab791de032d758aac P 39712c4e2cd146c295e69aedb6e6bdee: Deleting consensus metadata
I20260928 10:56:17.706724 24291 catalog_manager.cc:4841] TS 39712c4e2cd146c295e69aedb6e6bdee (127.5.227.129:36693): tablet fff6d38ff665411ab791de032d758aac (table test-table [id=314dd33f860c4f8b9af4f3cf0a1e7aac]) successfully deleted
I20260928 10:56:17.710069 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.657606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.5.227.129:36693)
0928 10:56:17.657864 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.657874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.676398 (+ 18524us) server_negotiation.cc:407] Connection header received
0928 10:56:17.676634 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.676643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.676783 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.676901 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:17.679331 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.681766 (+ 2435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.685684 (+ 3918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.686506 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.686667 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.700901 (+ 14234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:17.700917 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:17.700933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:17.701248 (+ 315us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:17.706441 (+ 5193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.707500 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.707508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.707511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.707678 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.708165 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.708175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.708177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.708600 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:17.708902 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.709428 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.709689 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 10:56:17.710968 24489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:17.817524 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 24366
I20260928 10:56:17.839169 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 24500
Sep 28 10:56:17 dist-test-slave-8w8g krb5kdc[24260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:56:17.860345 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.840239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.5.227.130:41497)
0928 10:56:17.840433 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.840437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.840459 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:56:17.840751 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.840756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.841011 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.841172 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:17.842713 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.843493 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.845272 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.845867 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.845990 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.851162 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:17.851187 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:17.851205 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:17.851256 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:17.855735 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.857145 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.857156 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.857159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.857341 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.857892 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.857896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.857899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.858425 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:17.858721 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.859671 (+ 950us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.860031 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:56:17.861202 24623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:17.959079 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 24500
I20260928 10:56:17.977583 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 24634
Sep 28 10:56:17 dist-test-slave-8w8g krb5kdc[24260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:56:17.998288 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:17.978534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.5.227.131:41971)
0928 10:56:17.978716 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:17.978720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:17.978740 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:56:17.978800 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:17.978803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:17.978877 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:17.978972 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:17.980820 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.981695 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.983232 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:17.983824 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:17.984007 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:17.989390 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:17.989408 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:17.989427 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:17.989475 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:17.993721 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.994934 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.994948 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.994953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.995174 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:17.996186 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:17.996196 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:17.996198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:17.996765 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:17.997049 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:17.997608 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 10:56:17.997834 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:56:17.999285 24757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:18.102680 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 24634
I20260928 10:56:18.122673 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 24273
I20260928 10:56:18.124433 24335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:18.249018 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 24273
2026-09-28T10:56:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T10:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:18Z Disabled control of system clock
I20260928 10:56:18.340233 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41171
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:56:18.735558 24816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:18.735988 24816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:18.736079 24816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:18.743667 24816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:18.743870 24816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:18.743922 24816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:18.743999 24816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:18.744048 24816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:18.753882 24816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:41171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:18.755597 24816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:18.757660 24816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:18.772171 24821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:18.772152 24824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:18.772676 24822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:18.773639 24816 server_base.cc:1034] running on GCE node
I20260928 10:56:18.774605 24816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:18.776540 24816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:18.777860 24816 hybrid_clock.cc:648] HybridClock initialized: now 1790592978777750 us; error 93 us; skew 500 ppm
I20260928 10:56:18.778704 24816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:18.782003 24816 webserver.cc:460] Webserver started at http://127.5.227.190:42839/ using document root <none> and password file <none>
I20260928 10:56:18.782876 24816 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:18.783255 24816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:18.783644 24816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:18.787410 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "86366f061f924b2ba8d19b8c403e5369"
format_stamp: "Formatted at 2026-09-28 10:56:18 on dist-test-slave-8w8g"
server_key: "04268772c9aec302d7751c7c85335821"
server_key_iv: "6e22bb259412d4cb6b55abcb753f012f"
server_key_version: "encryptionkey@0"
I20260928 10:56:18.788658 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "86366f061f924b2ba8d19b8c403e5369"
format_stamp: "Formatted at 2026-09-28 10:56:18 on dist-test-slave-8w8g"
server_key: "04268772c9aec302d7751c7c85335821"
server_key_iv: "6e22bb259412d4cb6b55abcb753f012f"
server_key_version: "encryptionkey@0"
I20260928 10:56:18.795042 24816 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:56:18.799463 24830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:18.801733 24816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:56:18.802037 24816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "86366f061f924b2ba8d19b8c403e5369"
format_stamp: "Formatted at 2026-09-28 10:56:18 on dist-test-slave-8w8g"
server_key: "04268772c9aec302d7751c7c85335821"
server_key_iv: "6e22bb259412d4cb6b55abcb753f012f"
server_key_version: "encryptionkey@0"
I20260928 10:56:18.802335 24816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:18.836479 24816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:18.837405 24816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:18.837723 24816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:18.859004 24816 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:41171
I20260928 10:56:18.859021 24881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:41171 every 8 connection(s)
I20260928 10:56:18.860522 24816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:18.864622 24882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:18.870832 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 24816
I20260928 10:56:18.871042 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:18.871562 6030 external_mini_cluster.cc:1428] Setting key 2e0cad58e384e928fd5f3656af19720b
I20260928 10:56:18.878302 24882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369: Bootstrap starting.
I20260928 10:56:18.881804 24882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:18.882941 24882 log.cc:824] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:18.886410 24882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369: No bootstrap required, opened a new log
I20260928 10:56:18.888749 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:18.874831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.5.227.190:41171)
0928 10:56:18.876429 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:18.876442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:18.876491 (+ 49us) server_negotiation.cc:407] Connection header received
0928 10:56:18.878332 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:18.878358 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:18.878967 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:18.879429 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:18.880111 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:18.881499 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:18.883041 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:18.884163 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:18.884505 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:18.885977 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:18.885986 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:18.886001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:18.886682 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:18.886776 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:18.886802 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:18.886968 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:18.887209 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 10:56:18.887574 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":552,"threads_started":1}
I20260928 10:56:18.897148 24882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86366f061f924b2ba8d19b8c403e5369" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41171 } }
I20260928 10:56:18.897766 24882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:18.897888 24882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86366f061f924b2ba8d19b8c403e5369, State: Initialized, Role: FOLLOWER
I20260928 10:56:18.898595 24882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86366f061f924b2ba8d19b8c403e5369" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41171 } }
I20260928 10:56:18.898875 24882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:18.899044 24882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:18.899240 24882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:18.901651 24882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86366f061f924b2ba8d19b8c403e5369" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41171 } }
I20260928 10:56:18.902253 24882 leader_election.cc:302] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86366f061f924b2ba8d19b8c403e5369; no voters:
I20260928 10:56:18.903110 24882 leader_election.cc:288] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:18.903293 24887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:18.904255 24887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [term 1 LEADER]: Becoming Leader. State: Replica: 86366f061f924b2ba8d19b8c403e5369, State: Running, Role: LEADER
I20260928 10:56:18.905196 24887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86366f061f924b2ba8d19b8c403e5369" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41171 } }
I20260928 10:56:18.906090 24882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:18.908738 24889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86366f061f924b2ba8d19b8c403e5369. Latest consensus state: current_term: 1 leader_uuid: "86366f061f924b2ba8d19b8c403e5369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86366f061f924b2ba8d19b8c403e5369" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41171 } } }
I20260928 10:56:18.909437 24889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:18.909906 24888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86366f061f924b2ba8d19b8c403e5369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86366f061f924b2ba8d19b8c403e5369" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41171 } } }
I20260928 10:56:18.910491 24888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:18.913465 24895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:18.919132 24895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:18.929862 24895 catalog_manager.cc:1347] Generated new cluster ID: ce3624fdcbe0478ea5402640126aab07
I20260928 10:56:18.930070 24895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:18.955063 24895 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:18.956820 24895 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:18.968528 24895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369: Generated new TSK 0
I20260928 10:56:18.969537 24895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:18.985683 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41171
--builtin_ntp_servers=127.5.227.148:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:56:19.400993 24906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:19.401394 24906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:19.401571 24906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:19.411834 24906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:19.412014 24906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:19.412215 24906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:56:19.422971 24906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:19.424527 24906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:19.427239 24906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:19.447255 24912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:19.447276 24911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:19.447512 24914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:19.448822 24906 server_base.cc:1034] running on GCE node
I20260928 10:56:19.449739 24906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:19.450986 24906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:19.454409 24906 hybrid_clock.cc:648] HybridClock initialized: now 1790592979454360 us; error 101 us; skew 500 ppm
I20260928 10:56:19.455075 24906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:19.458514 24906 webserver.cc:460] Webserver started at http://127.5.227.129:41443/ using document root <none> and password file <none>
I20260928 10:56:19.459370 24906 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:19.459475 24906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:19.459856 24906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:19.463126 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "d2ec6705d71143ca820002141c22157f"
format_stamp: "Formatted at 2026-09-28 10:56:19 on dist-test-slave-8w8g"
server_key: "7baca3c45314299d47ffb02c67084a78"
server_key_iv: "a3a5a61e6941867b1e1430dadae89350"
server_key_version: "encryptionkey@0"
I20260928 10:56:19.464449 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "d2ec6705d71143ca820002141c22157f"
format_stamp: "Formatted at 2026-09-28 10:56:19 on dist-test-slave-8w8g"
server_key: "7baca3c45314299d47ffb02c67084a78"
server_key_iv: "a3a5a61e6941867b1e1430dadae89350"
server_key_version: "encryptionkey@0"
I20260928 10:56:19.471662 24906 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 10:56:19.476264 24920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:19.478216 24906 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:56:19.478394 24906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "d2ec6705d71143ca820002141c22157f"
format_stamp: "Formatted at 2026-09-28 10:56:19 on dist-test-slave-8w8g"
server_key: "7baca3c45314299d47ffb02c67084a78"
server_key_iv: "a3a5a61e6941867b1e1430dadae89350"
server_key_version: "encryptionkey@0"
I20260928 10:56:19.478899 24906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:19.513829 24906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:19.514752 24906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:19.515571 24906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:19.516830 24906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:19.518978 24906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:19.519116 24906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:19.519209 24906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:19.519273 24906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:19.558852 24906 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:36571
I20260928 10:56:19.558979 25032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:36571 every 8 connection(s)
I20260928 10:56:19.560290 24906 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:56:19.561767 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 24906
I20260928 10:56:19.561980 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:56:19.562623 6030 external_mini_cluster.cc:1428] Setting key 518689ee793e03b76dd59a064d226052
I20260928 10:56:19.566591 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41171
--builtin_ntp_servers=127.5.227.148:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:56:19.582433 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:19.564019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:48545 (local address 127.5.227.190:41171)
0928 10:56:19.564521 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:19.564528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:19.566262 (+ 1734us) server_negotiation.cc:407] Connection header received
0928 10:56:19.569882 (+ 3620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:19.569892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:19.570008 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:19.570149 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:19.573256 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:19.574930 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:19.578167 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:19.578983 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:19.579123 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:19.581425 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:19.581431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:19.581434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:19.581459 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:19.581530 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:19.581545 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:19.581628 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:19.582048 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 10:56:19.582154 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260928 10:56:19.583954 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:19.564532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41171 (local address 127.5.227.129:48545)
0928 10:56:19.566046 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:19.566100 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:19.569140 (+ 3040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:19.570730 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:19.570749 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:19.571000 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:19.572712 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:19.572763 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:19.575224 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:19.575233 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:19.577907 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:19.577928 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:19.579304 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:19.579314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:19.579905 (+ 591us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:19.580870 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:19.580876 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:19.580945 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:19.581078 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:19.581674 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:19.581689 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:19.581702 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 10:56:19.581939 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 10:56:19.582440 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":548,"threads_started":1}
I20260928 10:56:19.586498 25033 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41171
I20260928 10:56:19.586968 25033 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:19.588328 25033 heartbeater.cc:511] Master 127.5.227.190:41171 requested a full tablet report, sending...
I20260928 10:56:19.591384 24847 ts_manager.cc:194] Registered new tserver with Master: d2ec6705d71143ca820002141c22157f (127.5.227.129:36571)
I20260928 10:56:19.593652 24847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.129:48545
W20260928 10:56:19.949793 25037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:19.950111 25037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:19.950258 25037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:19.958045 25037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:19.958160 25037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:19.958328 25037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:56:19.968035 25037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:19.969449 25037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:19.971318 25037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:19.988200 25043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:19.988210 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:19.988979 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:19.989629 25037 server_base.cc:1034] running on GCE node
I20260928 10:56:19.990331 25037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:19.991446 25037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:19.992740 25037 hybrid_clock.cc:648] HybridClock initialized: now 1790592979992691 us; error 95 us; skew 500 ppm
I20260928 10:56:19.993139 25037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:19.996408 25037 webserver.cc:460] Webserver started at http://127.5.227.130:38387/ using document root <none> and password file <none>
I20260928 10:56:19.997280 25037 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:19.997442 25037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:19.997864 25037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:20.002306 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "3af4dfcb31864b7387ccb6d178c7da98"
format_stamp: "Formatted at 2026-09-28 10:56:19 on dist-test-slave-8w8g"
server_key: "7078d53f2c174750a98851fc1c29319c"
server_key_iv: "9035010f5ad8c0454e7fa9f1cdf3c5c5"
server_key_version: "encryptionkey@0"
I20260928 10:56:20.003532 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "3af4dfcb31864b7387ccb6d178c7da98"
format_stamp: "Formatted at 2026-09-28 10:56:19 on dist-test-slave-8w8g"
server_key: "7078d53f2c174750a98851fc1c29319c"
server_key_iv: "9035010f5ad8c0454e7fa9f1cdf3c5c5"
server_key_version: "encryptionkey@0"
I20260928 10:56:20.010802 25037 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260928 10:56:20.015818 25051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:20.018245 25037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:56:20.018445 25037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "3af4dfcb31864b7387ccb6d178c7da98"
format_stamp: "Formatted at 2026-09-28 10:56:19 on dist-test-slave-8w8g"
server_key: "7078d53f2c174750a98851fc1c29319c"
server_key_iv: "9035010f5ad8c0454e7fa9f1cdf3c5c5"
server_key_version: "encryptionkey@0"
I20260928 10:56:20.018898 25037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:20.058905 25037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:20.059903 25037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:20.060248 25037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:20.061478 25037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:20.063462 25037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:20.063637 25037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:20.063828 25037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:20.063920 25037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:20.103551 25037 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:44255
I20260928 10:56:20.103611 25163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:44255 every 8 connection(s)
I20260928 10:56:20.104974 25037 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:56:20.114892 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25037
I20260928 10:56:20.115129 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:56:20.115586 6030 external_mini_cluster.cc:1428] Setting key 5a52ff15063d6d7a83a27bd636031bb6
I20260928 10:56:20.119163 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41171
--builtin_ntp_servers=127.5.227.148:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:56:20.120807 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.108208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:38167 (local address 127.5.227.190:41171)
0928 10:56:20.108827 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.108832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.110103 (+ 1271us) server_negotiation.cc:407] Connection header received
0928 10:56:20.112652 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.112658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.112777 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.112948 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:20.114252 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.115042 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.117045 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.117508 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.117654 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.119582 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:20.119591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:20.119598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:20.119626 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:20.119679 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:20.119699 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:20.119820 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.120176 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.120308 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":387,"threads_started":1}
I20260928 10:56:20.121917 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.108876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41171 (local address 127.5.227.130:38167)
0928 10:56:20.109906 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:20.109952 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:20.112147 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:20.113218 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:20.113228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:20.113339 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:20.114033 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.114052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.115256 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.115266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.116880 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.116896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.117772 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.117783 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.118129 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.119087 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:20.119094 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:20.119161 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:20.119294 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:20.119909 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:20.119919 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:20.119927 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:56:20.120135 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 10:56:20.120646 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":306,"threads_started":1}
I20260928 10:56:20.124127 25164 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41171
I20260928 10:56:20.124737 25164 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:20.126024 25164 heartbeater.cc:511] Master 127.5.227.190:41171 requested a full tablet report, sending...
I20260928 10:56:20.128005 24847 ts_manager.cc:194] Registered new tserver with Master: 3af4dfcb31864b7387ccb6d178c7da98 (127.5.227.130:44255)
I20260928 10:56:20.128944 24847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.130:38167
W20260928 10:56:20.501241 25168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:20.501590 25168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:20.501696 25168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:20.510761 25168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:20.510897 25168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:20.511014 25168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:56:20.523202 25168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:20.524947 25168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:20.527035 25168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:20.547401 25173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:20.547771 25174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:20.547646 25168 server_base.cc:1034] running on GCE node
W20260928 10:56:20.548224 25176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:20.549212 25168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:20.550601 25168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:20.551890 25168 hybrid_clock.cc:648] HybridClock initialized: now 1790592980551831 us; error 50 us; skew 500 ppm
I20260928 10:56:20.552274 25168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:20.555205 25168 webserver.cc:460] Webserver started at http://127.5.227.131:38345/ using document root <none> and password file <none>
I20260928 10:56:20.556180 25168 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:20.556335 25168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:20.556715 25168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:20.561291 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "fa2bfa1c22cb4b5b8070feb7d457f513"
format_stamp: "Formatted at 2026-09-28 10:56:20 on dist-test-slave-8w8g"
server_key: "223c9882fd01c39a25fe3a53c214a9fc"
server_key_iv: "b5a096052e7011f9789acbde135a1fe8"
server_key_version: "encryptionkey@0"
I20260928 10:56:20.562846 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "fa2bfa1c22cb4b5b8070feb7d457f513"
format_stamp: "Formatted at 2026-09-28 10:56:20 on dist-test-slave-8w8g"
server_key: "223c9882fd01c39a25fe3a53c214a9fc"
server_key_iv: "b5a096052e7011f9789acbde135a1fe8"
server_key_version: "encryptionkey@0"
I20260928 10:56:20.571298 25168 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 10:56:20.575990 25182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:20.577855 25168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:20.578087 25168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "fa2bfa1c22cb4b5b8070feb7d457f513"
format_stamp: "Formatted at 2026-09-28 10:56:20 on dist-test-slave-8w8g"
server_key: "223c9882fd01c39a25fe3a53c214a9fc"
server_key_iv: "b5a096052e7011f9789acbde135a1fe8"
server_key_version: "encryptionkey@0"
I20260928 10:56:20.578346 25168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:20.598667 25033 heartbeater.cc:503] Master 127.5.227.190:41171 was elected leader, sending a full tablet report...
I20260928 10:56:20.602355 25168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:20.603487 25168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:20.603910 25168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:20.605062 25168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:20.607383 25168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:20.607517 25168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:20.607666 25168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:20.607791 25168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:20.653887 25168 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:34829
I20260928 10:56:20.654054 25294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:34829 every 8 connection(s)
I20260928 10:56:20.655380 25168 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:56:20.662899 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25168
I20260928 10:56:20.663137 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:56:20.663648 6030 external_mini_cluster.cc:1428] Setting key 0816b2a8d72be9b00fd41079e83e83d6
I20260928 10:56:20.673120 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.658888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:33377 (local address 127.5.227.190:41171)
0928 10:56:20.659502 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.659508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.661662 (+ 2154us) server_negotiation.cc:407] Connection header received
0928 10:56:20.664130 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.664138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.664262 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.664427 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:20.666069 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.666937 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.669512 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.669963 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.670216 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.672077 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:20.672092 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:20.672095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:20.672119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:20.672162 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:20.672174 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:20.672286 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.672558 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.672680 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":295,"threads_started":1}
I20260928 10:56:20.674367 25298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.659869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41171 (local address 127.5.227.131:33377)
0928 10:56:20.661340 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:20.661383 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:20.663665 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:20.664732 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:20.664743 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:20.664871 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:20.665837 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.665864 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.667137 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.667142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.669003 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.669021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.670338 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.670348 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.670770 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.671590 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:20.671593 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:20.671639 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:20.671822 (+ 183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:20.672289 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:20.672303 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:20.672314 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:56:20.672529 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 10:56:20.673047 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":551,"threads_started":1}
I20260928 10:56:20.677011 25295 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41171
I20260928 10:56:20.677595 25295 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:20.678525 25295 heartbeater.cc:511] Master 127.5.227.190:41171 requested a full tablet report, sending...
I20260928 10:56:20.680431 24847 ts_manager.cc:194] Registered new tserver with Master: fa2bfa1c22cb4b5b8070feb7d457f513 (127.5.227.131:34829)
I20260928 10:56:20.681267 24847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.131:33377
I20260928 10:56:20.693678 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:56:20.707988 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.699314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.5.227.190:41171)
0928 10:56:20.699519 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.699524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.700114 (+ 590us) server_negotiation.cc:407] Connection header received
0928 10:56:20.700239 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.700244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.700319 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.700432 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:20.701144 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.702026 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.704032 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.704627 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.704792 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.706313 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:20.706319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:20.706322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:20.706345 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:20.706381 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:20.706394 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:20.706820 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.707613 (+ 793us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.707669 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:56:20.713939 24847 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:56:20.722265 24847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:56:20.764793 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.738503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44255 (local address 127.0.0.1:56994)
0928 10:56:20.739524 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:20.739590 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:20.739853 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:20.757287 (+ 17434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:20.757292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:20.757322 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:20.757708 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.757721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.759757 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.759764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.761753 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.761775 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.763359 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.763366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.763619 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.763688 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 10:56:20.763861 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 10:56:20.764314 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":1,"thread_start_us":883,"threads_started":1}
I20260928 10:56:20.765499 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.739011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:34829 (local address 127.0.0.1:45528)
0928 10:56:20.740535 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:20.740552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:20.740652 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:20.756702 (+ 16050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:20.756708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:20.756740 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:20.757315 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.757330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.760718 (+ 3388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.760729 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.762235 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.762250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.764134 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.764142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.764369 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.764412 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:56:20.764499 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 10:56:20.764639 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"mutex_wait_us":1,"thread_start_us":1213,"threads_started":1}
I20260928 10:56:20.765594 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.739830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.5.227.131:34829)
0928 10:56:20.755637 (+ 15807us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.755651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.755688 (+ 37us) server_negotiation.cc:407] Connection header received
0928 10:56:20.755835 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.755846 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.756296 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.757656 (+ 1360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:20.757932 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.760521 (+ 2589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.762471 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.763694 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.763960 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.764294 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.764895 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.765083 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15594,"thread_start_us":15795,"threads_started":1}
I20260928 10:56:20.765448 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.739170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.5.227.130:44255)
0928 10:56:20.756430 (+ 17260us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.756440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.756471 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:56:20.756702 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.756710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.756962 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.757294 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:20.757979 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.759551 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.762021 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.763113 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.764429 (+ 1316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.764692 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.764773 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.765101 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17056,"thread_start_us":16798,"threads_started":1}
I20260928 10:56:20.766924 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.740628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33072)
0928 10:56:20.753875 (+ 13247us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:20.753902 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:20.754090 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:20.756481 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:20.756490 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:20.756536 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:20.757013 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.757026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.758708 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.758715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.762886 (+ 4171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.762901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.765997 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.766005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.766180 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.766213 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 10:56:20.766313 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 10:56:20.766450 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13075,"mutex_wait_us":105,"thread_start_us":12774,"threads_started":1}
I20260928 10:56:20.767868 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.741191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.5.227.129:36571)
0928 10:56:20.754278 (+ 13087us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.754904 (+ 626us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.755410 (+ 506us) server_negotiation.cc:407] Connection header received
0928 10:56:20.755677 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.755693 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.756147 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.756437 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:20.757200 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.758546 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.763150 (+ 4604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.765259 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.765406 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.767191 (+ 1785us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.767291 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.767486 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12857,"thread_start_us":13229,"threads_started":1}
I20260928 10:56:20.771337 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 0348761f407341d4832532e734cc4290 (DEFAULT_TABLE table=test-table [id=235dba1f51ae4751850a7a5e8e1403cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:20.771355 25099 tablet_service.cc:1467] Processing CreateTablet for tablet 0348761f407341d4832532e734cc4290 (DEFAULT_TABLE table=test-table [id=235dba1f51ae4751850a7a5e8e1403cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:20.771162 25230 tablet_service.cc:1467] Processing CreateTablet for tablet 0348761f407341d4832532e734cc4290 (DEFAULT_TABLE table=test-table [id=235dba1f51ae4751850a7a5e8e1403cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:20.772691 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0348761f407341d4832532e734cc4290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:20.772761 25099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0348761f407341d4832532e734cc4290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:20.772760 25230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0348761f407341d4832532e734cc4290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:20.785913 25314 tablet_bootstrap.cc:492] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: Bootstrap starting.
I20260928 10:56:20.789438 25314 tablet_bootstrap.cc:654] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:20.790429 25315 tablet_bootstrap.cc:492] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Bootstrap starting.
I20260928 10:56:20.790923 25314 log.cc:824] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:20.793195 25316 tablet_bootstrap.cc:492] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Bootstrap starting.
I20260928 10:56:20.795650 25315 tablet_bootstrap.cc:654] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:20.797135 25315 log.cc:824] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:20.798555 25316 tablet_bootstrap.cc:654] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:20.798770 25314 tablet_bootstrap.cc:492] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: No bootstrap required, opened a new log
I20260928 10:56:20.799136 25314 ts_tablet_manager.cc:1397] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260928 10:56:20.800166 25316 log.cc:824] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:20.801182 25315 tablet_bootstrap.cc:492] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: No bootstrap required, opened a new log
I20260928 10:56:20.801518 25315 ts_tablet_manager.cc:1397] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260928 10:56:20.803887 25316 tablet_bootstrap.cc:492] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: No bootstrap required, opened a new log
I20260928 10:56:20.804293 25316 ts_tablet_manager.cc:1397] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260928 10:56:20.808439 25314 raft_consensus.cc:348] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.808835 25314 raft_consensus.cc:374] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:20.808904 25314 raft_consensus.cc:729] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ec6705d71143ca820002141c22157f, State: Initialized, Role: FOLLOWER
I20260928 10:56:20.809574 25314 consensus_queue.cc:260] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.813499 25314 ts_tablet_manager.cc:1428] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260928 10:56:20.813243 25315 raft_consensus.cc:348] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.814062 25315 raft_consensus.cc:374] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:20.814179 25315 raft_consensus.cc:729] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2bfa1c22cb4b5b8070feb7d457f513, State: Initialized, Role: FOLLOWER
W20260928 10:56:20.814903 25034 tablet.cc:2369] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:20.814909 25315 consensus_queue.cc:260] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.816417 25316 raft_consensus.cc:348] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.817039 25316 raft_consensus.cc:374] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:20.817173 25316 raft_consensus.cc:729] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af4dfcb31864b7387ccb6d178c7da98, State: Initialized, Role: FOLLOWER
I20260928 10:56:20.817994 25316 consensus_queue.cc:260] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.819350 25164 heartbeater.cc:503] Master 127.5.227.190:41171 was elected leader, sending a full tablet report...
I20260928 10:56:20.820427 25316 ts_tablet_manager.cc:1428] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Time spent starting tablet: real 0.016s user 0.005s sys 0.009s
I20260928 10:56:20.821736 25295 heartbeater.cc:503] Master 127.5.227.190:41171 was elected leader, sending a full tablet report...
I20260928 10:56:20.822707 25315 ts_tablet_manager.cc:1428] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Time spent starting tablet: real 0.021s user 0.014s sys 0.007s
W20260928 10:56:20.857476 25165 tablet.cc:2369] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:56:20.907792 25296 tablet.cc:2369] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:20.941530 25320 raft_consensus.cc:482] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:56:20.941907 25320 raft_consensus.cc:504] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.943660 25320 leader_election.cc:288] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3af4dfcb31864b7387ccb6d178c7da98 (127.5.227.130:44255), fa2bfa1c22cb4b5b8070feb7d457f513 (127.5.227.131:34829)
I20260928 10:56:20.951215 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.944234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:36547 (local address 127.5.227.130:44255)
0928 10:56:20.944445 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.944454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.945305 (+ 851us) server_negotiation.cc:407] Connection header received
0928 10:56:20.945521 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.945525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.945605 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.945711 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:20.946426 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.947263 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.949463 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.950223 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.950410 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.950485 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.950787 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.950928 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:56:20.951205 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.944208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:58193 (local address 127.5.227.131:34829)
0928 10:56:20.944420 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.944429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.945231 (+ 802us) server_negotiation.cc:407] Connection header received
0928 10:56:20.945448 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.945455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.945657 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.945791 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:20.946525 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.947241 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.949449 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.950249 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.950444 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.950516 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:20.950787 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 10:56:20.950930 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:56:20.951352 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.944261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:44255 (local address 127.5.227.129:36547)
0928 10:56:20.945214 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:20.945229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:20.945351 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:20.945728 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:20.945732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:20.945756 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:20.946229 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.946242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.947418 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.947427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.949221 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.949234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.950428 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.950435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.950610 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.950638 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 10:56:20.950755 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 10:56:20.950932 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":118,"thread_start_us":470,"threads_started":1}
I20260928 10:56:20.951851 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.944239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:34829 (local address 127.5.227.129:58193)
0928 10:56:20.945108 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:20.945123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:20.945313 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:20.945805 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:20.945809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:20.945903 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:20.946329 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.946338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.947385 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.947391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.949209 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:20.949223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.950464 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:20.950471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:20.950621 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:20.950644 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 10:56:20.950748 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 10:56:20.950964 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":152,"spinlock_wait_cycles":28544,"thread_start_us":475,"threads_started":1}
I20260928 10:56:20.952610 25119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0348761f407341d4832532e734cc4290" candidate_uuid: "d2ec6705d71143ca820002141c22157f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af4dfcb31864b7387ccb6d178c7da98" is_pre_election: true
I20260928 10:56:20.952750 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0348761f407341d4832532e734cc4290" candidate_uuid: "d2ec6705d71143ca820002141c22157f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" is_pre_election: true
I20260928 10:56:20.953275 25250 raft_consensus.cc:2454] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ec6705d71143ca820002141c22157f in term 0.
I20260928 10:56:20.953487 25119 raft_consensus.cc:2454] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ec6705d71143ca820002141c22157f in term 0.
I20260928 10:56:20.953984 24924 leader_election.cc:302] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2ec6705d71143ca820002141c22157f, fa2bfa1c22cb4b5b8070feb7d457f513; no voters:
I20260928 10:56:20.954457 25320 raft_consensus.cc:2781] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:56:20.954662 25320 raft_consensus.cc:482] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:56:20.954778 25320 raft_consensus.cc:3037] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:20.957055 25320 raft_consensus.cc:504] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.957723 25320 leader_election.cc:288] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [CANDIDATE]: Term 1 election: Requested vote from peers 3af4dfcb31864b7387ccb6d178c7da98 (127.5.227.130:44255), fa2bfa1c22cb4b5b8070feb7d457f513 (127.5.227.131:34829)
I20260928 10:56:20.958316 25119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0348761f407341d4832532e734cc4290" candidate_uuid: "d2ec6705d71143ca820002141c22157f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af4dfcb31864b7387ccb6d178c7da98"
I20260928 10:56:20.958338 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0348761f407341d4832532e734cc4290" candidate_uuid: "d2ec6705d71143ca820002141c22157f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513"
I20260928 10:56:20.958578 25119 raft_consensus.cc:3037] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:20.958624 25250 raft_consensus.cc:3037] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:20.961155 25119 raft_consensus.cc:2454] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ec6705d71143ca820002141c22157f in term 1.
I20260928 10:56:20.961151 25250 raft_consensus.cc:2454] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ec6705d71143ca820002141c22157f in term 1.
I20260928 10:56:20.961728 24921 leader_election.cc:302] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af4dfcb31864b7387ccb6d178c7da98, d2ec6705d71143ca820002141c22157f; no voters:
I20260928 10:56:20.962122 25320 raft_consensus.cc:2781] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:20.962644 25320 raft_consensus.cc:686] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 1 LEADER]: Becoming Leader. State: Replica: d2ec6705d71143ca820002141c22157f, State: Running, Role: LEADER
I20260928 10:56:20.963151 25320 consensus_queue.cc:237] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } }
I20260928 10:56:20.968407 24846 catalog_manager.cc:5495] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f reported cstate change: term changed from 0 to 1, leader changed from <none> to d2ec6705d71143ca820002141c22157f (127.5.227.129). New cstate: current_term: 1 leader_uuid: "d2ec6705d71143ca820002141c22157f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2ec6705d71143ca820002141c22157f" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 36571 } health_report { overall_health: HEALTHY } } }
I20260928 10:56:21.003088 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:20.996257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.5.227.129:36571)
0928 10:56:20.996478 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:20.996483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:20.996636 (+ 153us) server_negotiation.cc:407] Connection header received
0928 10:56:20.996812 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:20.996817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:20.996939 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:20.997172 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:56:20.997621 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:20.998535 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.000189 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:21.000679 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.000789 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:21.001523 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:56:21.002093 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:56:21.002569 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:21.002732 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 10:56:21.002796 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:56:21.013760 25250 raft_consensus.cc:1261] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 1 FOLLOWER]: Refusing update from remote peer d2ec6705d71143ca820002141c22157f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:21.013760 25119 raft_consensus.cc:1261] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 1 FOLLOWER]: Refusing update from remote peer d2ec6705d71143ca820002141c22157f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:21.014901 25325 consensus_queue.cc:1035] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa2bfa1c22cb4b5b8070feb7d457f513" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 34829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:21.015388 25320 consensus_queue.cc:1035] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af4dfcb31864b7387ccb6d178c7da98" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:21.048125 24847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:21.048461 24847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:21.052765 24847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86366f061f924b2ba8d19b8c403e5369: Sending DeleteTablet for 3 replicas of tablet 0348761f407341d4832532e734cc4290
I20260928 10:56:21.054072 25099 tablet_service.cc:1514] Processing DeleteTablet for tablet 0348761f407341d4832532e734cc4290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:21 UTC) from {username='slave'} at 127.0.0.1:56994
I20260928 10:56:21.054214 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet 0348761f407341d4832532e734cc4290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:21 UTC) from {username='slave'} at 127.0.0.1:33072
I20260928 10:56:21.054306 25230 tablet_service.cc:1514] Processing DeleteTablet for tablet 0348761f407341d4832532e734cc4290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:21 UTC) from {username='slave'} at 127.0.0.1:45528
I20260928 10:56:21.058548 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 24906
I20260928 10:56:21.074461 25338 tablet_replica.cc:330] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: stopping tablet replica
I20260928 10:56:21.075053 25338 raft_consensus.cc:2229] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:21.075611 25338 raft_consensus.cc:2258] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:21.076289 25339 tablet_replica.cc:330] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f: stopping tablet replica
I20260928 10:56:21.077747 25339 raft_consensus.cc:2229] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:56:21.080276 25339 raft_consensus.cc:2258] T 0348761f407341d4832532e734cc4290 P d2ec6705d71143ca820002141c22157f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:21.083647 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.062246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.5.227.129:36571)
0928 10:56:21.062982 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:21.062989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:21.071211 (+ 8222us) server_negotiation.cc:407] Connection header received
0928 10:56:21.071800 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:21.071810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:21.071959 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:21.072063 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:21.072984 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:21.074377 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.077992 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:21.078685 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.078792 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:21.081118 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:21.081129 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:21.081148 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:21.081986 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:21.082053 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:21.082080 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:21.082169 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:21.082926 (+ 757us) server_negotiation.cc:299] Negotiation successful
0928 10:56:21.083091 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600}
I20260928 10:56:21.085654 25028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:21.086024 25338 ts_tablet_manager.cc:1905] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:21.086570 25340 tablet_replica.cc:330] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: stopping tablet replica
I20260928 10:56:21.087100 25340 raft_consensus.cc:2229] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:21.087723 25340 raft_consensus.cc:2258] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:21.090134 25340 ts_tablet_manager.cc:1905] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:21.093503 25338 ts_tablet_manager.cc:1918] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:21.093791 25338 log.cc:1196] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/0348761f407341d4832532e734cc4290
I20260928 10:56:21.094489 25338 ts_tablet_manager.cc:1939] T 0348761f407341d4832532e734cc4290 P 3af4dfcb31864b7387ccb6d178c7da98: Deleting consensus metadata
I20260928 10:56:21.097280 25340 ts_tablet_manager.cc:1918] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:21.097208 24831 catalog_manager.cc:4841] TS 3af4dfcb31864b7387ccb6d178c7da98 (127.5.227.130:44255): tablet 0348761f407341d4832532e734cc4290 (table test-table [id=235dba1f51ae4751850a7a5e8e1403cd]) successfully deleted
I20260928 10:56:21.097489 25340 log.cc:1196] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/0348761f407341d4832532e734cc4290
I20260928 10:56:21.098070 25340 ts_tablet_manager.cc:1939] T 0348761f407341d4832532e734cc4290 P fa2bfa1c22cb4b5b8070feb7d457f513: Deleting consensus metadata
I20260928 10:56:21.100281 24834 catalog_manager.cc:4841] TS fa2bfa1c22cb4b5b8070feb7d457f513 (127.5.227.131:34829): tablet 0348761f407341d4832532e734cc4290 (table test-table [id=235dba1f51ae4751850a7a5e8e1403cd]) successfully deleted
I20260928 10:56:21.220135 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 24906
I20260928 10:56:21.240641 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25037
W20260928 10:56:21.240670 24834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.227.129:36571 (error 108)
I20260928 10:56:21.242482 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.242005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33106)
0928 10:56:21.242210 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:21.242319 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 10:56:21.242885 24834 catalog_manager.cc:4584] TS d2ec6705d71143ca820002141c22157f (127.5.227.129:36571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0348761f407341d4832532e734cc4290: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
I20260928 10:56:21.249219 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.242040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.5.227.130:44255)
0928 10:56:21.242281 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:21.242286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:21.242302 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:56:21.242356 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:21.242360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:21.242455 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:21.242578 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:21.243058 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:21.243988 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.245365 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:21.245866 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.245999 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:21.247202 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:21.247211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:21.247225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:21.248385 (+ 1160us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:21.248444 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:21.248467 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:21.248586 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:21.248793 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 10:56:21.248891 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:56:21.250046 25159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:21.296697 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.296147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33116)
0928 10:56:21.296398 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:21.296512 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 10:56:21.297943 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.297477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33120)
0928 10:56:21.297644 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:21.297744 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 10:56:21.345109 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25037
I20260928 10:56:21.358918 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.358245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33130)
0928 10:56:21.358536 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:21.358671 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 10:56:21.360585 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.359994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33144)
0928 10:56:21.360241 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:21.360362 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 10:56:21.364975 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25168
I20260928 10:56:21.373018 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.365911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.5.227.131:34829)
0928 10:56:21.366092 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:21.366097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:21.366116 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:56:21.366248 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:21.366252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:21.366339 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:21.366446 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:21.366849 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:21.367619 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.369387 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:21.369953 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:21.370062 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:21.371273 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:21.371284 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:21.371299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:21.372241 (+ 942us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:21.372294 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:21.372316 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:21.372420 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:21.372635 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 10:56:21.372734 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 10:56:21.373894 25290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:21.441434 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.440819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33160)
0928 10:56:21.441106 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:21.441221 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 10:56:21.443099 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:21.442520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:36571 (local address 127.0.0.1:33166)
0928 10:56:21.442791 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:21.442939 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 10:56:21.470726 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25168
I20260928 10:56:21.488153 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 24816
I20260928 10:56:21.489382 24877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:21.605923 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 24816
2026-09-28T10:56:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3349 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:21 dist-test-slave-8w8g krb5kdc[25349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:21 dist-test-slave-8w8g krb5kdc[25349](info): set up 2 sockets
Sep 28 10:56:21 dist-test-slave-8w8g krb5kdc[25349](info): commencing operation
krb5kdc: starting...
W20260928 10:56:22.780435 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:22 dist-test-slave-8w8g krb5kdc[25349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:56:23.077548 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41485
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:41485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:23.471488 25362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:23.471938 25362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:23.472131 25362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:23.480530 25362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:23.480736 25362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:23.480857 25362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:23.480963 25362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:23.481065 25362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:23.492522 25362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:41485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:41485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:23.494019 25362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:23.495615 25362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:23.509892 25368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:23.510010 25367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:23.510071 25370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:23.510826 25362 server_base.cc:1034] running on GCE node
I20260928 10:56:23.511780 25362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:23.513352 25362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:23.514665 25362 hybrid_clock.cc:648] HybridClock initialized: now 1790592983514587 us; error 62 us; skew 500 ppm
I20260928 10:56:23.515151 25362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:23 dist-test-slave-8w8g krb5kdc[25349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592983, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:23.524523 25362 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:56:23.526330 25362 webserver.cc:460] Webserver started at http://127.5.227.190:43153/ using document root <none> and password file <none>
I20260928 10:56:23.527146 25362 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:23.527307 25362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:23.527644 25362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:23.530956 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "748e83475f45448abcddd58e9fc52a48"
format_stamp: "Formatted at 2026-09-28 10:56:23 on dist-test-slave-8w8g"
server_key: "2e18bdad13bddc08e53eaae94f671ab2"
server_key_iv: "da83b448841e2f0eaf92eb8348900925"
server_key_version: "encryptionkey@0"
I20260928 10:56:23.531899 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "748e83475f45448abcddd58e9fc52a48"
format_stamp: "Formatted at 2026-09-28 10:56:23 on dist-test-slave-8w8g"
server_key: "2e18bdad13bddc08e53eaae94f671ab2"
server_key_iv: "da83b448841e2f0eaf92eb8348900925"
server_key_version: "encryptionkey@0"
I20260928 10:56:23.537866 25362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:56:23.542040 25377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:23.544065 25362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:23.544288 25362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "748e83475f45448abcddd58e9fc52a48"
format_stamp: "Formatted at 2026-09-28 10:56:23 on dist-test-slave-8w8g"
server_key: "2e18bdad13bddc08e53eaae94f671ab2"
server_key_iv: "da83b448841e2f0eaf92eb8348900925"
server_key_version: "encryptionkey@0"
I20260928 10:56:23.544520 25362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:23.576443 25362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:23.589123 25362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:23.589411 25362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:23.616761 25428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:41485 every 8 connection(s)
I20260928 10:56:23.616761 25362 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:41485
I20260928 10:56:23.619088 25362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:23.619695 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25362
I20260928 10:56:23.619974 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:23.620656 6030 external_mini_cluster.cc:1428] Setting key 043297873997f622cf1480c3654d3098
I20260928 10:56:23.625095 25429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:56:23 dist-test-slave-8w8g krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:23.641294 25429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48: Bootstrap starting.
I20260928 10:56:23.646135 25429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:23.648129 25429 log.cc:824] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:23.653214 25429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48: No bootstrap required, opened a new log
I20260928 10:56:23.654100 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:23.622767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.5.227.190:41485)
0928 10:56:23.623719 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:23.623766 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:23.623825 (+ 59us) server_negotiation.cc:407] Connection header received
0928 10:56:23.626798 (+ 2973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:23.626834 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:23.627612 (+ 778us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:23.628185 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:23.629726 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:23.632792 (+ 3066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:23.634315 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:23.635908 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:23.636128 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:23.642076 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:23.642088 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:23.642110 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:23.642166 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:23.647170 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:23.648341 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:23.648353 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:23.648362 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:23.648808 (+ 446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:23.649389 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:23.649393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:23.649395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:23.650658 (+ 1263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:23.650975 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:23.651711 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 10:56:23.652275 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":374,"threads_started":1}
I20260928 10:56:23.664197 25429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e83475f45448abcddd58e9fc52a48" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41485 } }
I20260928 10:56:23.664709 25429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:23.664793 25429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748e83475f45448abcddd58e9fc52a48, State: Initialized, Role: FOLLOWER
I20260928 10:56:23.665602 25429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e83475f45448abcddd58e9fc52a48" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41485 } }
I20260928 10:56:23.665887 25429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:23.666003 25429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:23.666218 25429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:23.669157 25429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e83475f45448abcddd58e9fc52a48" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41485 } }
I20260928 10:56:23.669801 25429 leader_election.cc:302] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 748e83475f45448abcddd58e9fc52a48; no voters:
I20260928 10:56:23.670791 25429 leader_election.cc:288] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:23.670930 25434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:23.671800 25434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [term 1 LEADER]: Becoming Leader. State: Replica: 748e83475f45448abcddd58e9fc52a48, State: Running, Role: LEADER
I20260928 10:56:23.672494 25434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e83475f45448abcddd58e9fc52a48" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41485 } }
I20260928 10:56:23.674371 25429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:23.675673 25435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "748e83475f45448abcddd58e9fc52a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e83475f45448abcddd58e9fc52a48" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41485 } } }
I20260928 10:56:23.675678 25436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 748e83475f45448abcddd58e9fc52a48. Latest consensus state: current_term: 1 leader_uuid: "748e83475f45448abcddd58e9fc52a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e83475f45448abcddd58e9fc52a48" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 41485 } } }
I20260928 10:56:23.676206 25435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:23.676206 25436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:23.680189 25439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:23.687214 25439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:23.699334 25439 catalog_manager.cc:1347] Generated new cluster ID: 0ed1f8876fb74fb886648721140245dd
I20260928 10:56:23.699476 25439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:23.721902 25439 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:23.723325 25439 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:23.732360 25439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48: Generated new TSK 0
I20260928 10:56:23.733655 25439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
Entry for principal kudu/127.5.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab.
I20260928 10:56:23.817467 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41485
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:24.206574 25455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:24.206897 25455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:24.207050 25455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:24.215960 25455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:24.216104 25455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:24.216317 25455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:56:24.228710 25455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.129.keytab
--principal=kudu/127.5.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:24.230186 25455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:24.231611 25455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:24.247392 25463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:24.247411 25460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:24.247713 25461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:24.248359 25455 server_base.cc:1034] running on GCE node
I20260928 10:56:24.249145 25455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:24.250264 25455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:24.251551 25455 hybrid_clock.cc:648] HybridClock initialized: now 1790592984251497 us; error 31 us; skew 500 ppm
I20260928 10:56:24.252029 25455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:24 dist-test-slave-8w8g krb5kdc[25349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592984, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:24.255986 25455 init.cc:377] Logged in from keytab as kudu/127.5.227.129@KRBTEST.COM (short username kudu)
I20260928 10:56:24.257822 25455 webserver.cc:460] Webserver started at http://127.5.227.129:45517/ using document root <none> and password file <none>
I20260928 10:56:24.258630 25455 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:24.258772 25455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:24.259131 25455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:24.262977 25455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "12615ca5f84f432aae463a08fb08dc4c"
format_stamp: "Formatted at 2026-09-28 10:56:24 on dist-test-slave-8w8g"
server_key: "4f7f2082417909a2b3b807c1cd0eb490"
server_key_iv: "064ee8166e422142426a1326060134b3"
server_key_version: "encryptionkey@0"
I20260928 10:56:24.264065 25455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "12615ca5f84f432aae463a08fb08dc4c"
format_stamp: "Formatted at 2026-09-28 10:56:24 on dist-test-slave-8w8g"
server_key: "4f7f2082417909a2b3b807c1cd0eb490"
server_key_iv: "064ee8166e422142426a1326060134b3"
server_key_version: "encryptionkey@0"
I20260928 10:56:24.270503 25455 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:56:24.275378 25470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:24.277585 25455 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260928 10:56:24.277860 25455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "12615ca5f84f432aae463a08fb08dc4c"
format_stamp: "Formatted at 2026-09-28 10:56:24 on dist-test-slave-8w8g"
server_key: "4f7f2082417909a2b3b807c1cd0eb490"
server_key_iv: "064ee8166e422142426a1326060134b3"
server_key_version: "encryptionkey@0"
I20260928 10:56:24.278123 25455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:24.291924 25455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:24.296804 25455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:24.297142 25455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:24.298292 25455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:24.299762 25455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:24.299870 25455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:24.300024 25455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:24.300104 25455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:24.341132 25455 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:40183
I20260928 10:56:24.341254 25582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:40183 every 8 connection(s)
I20260928 10:56:24.342968 25455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:56:24.350665 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25455
I20260928 10:56:24.350883 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:56:24.351472 6030 external_mini_cluster.cc:1428] Setting key 65550aa86b53238899922debe7249eba
Sep 28 10:56:24 dist-test-slave-8w8g krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592984, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.129@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:24.370442 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:24.346613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:43175 (local address 127.5.227.190:41485)
0928 10:56:24.347156 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:24.347162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:24.348386 (+ 1224us) server_negotiation.cc:407] Connection header received
0928 10:56:24.350611 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:24.350615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:24.350694 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:24.350848 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:24.352903 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.353965 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.357672 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.358100 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.358230 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:24.363585 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:24.363591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:24.363595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:24.363636 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:24.366722 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:24.367666 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:24.367671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:24.367674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:24.367909 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:24.368400 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:24.368405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:24.368409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:24.368714 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:24.368882 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:24.369805 (+ 923us) server_negotiation.cc:299] Negotiation successful
0928 10:56:24.370075 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":265,"threads_started":1}
I20260928 10:56:24.371500 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:24.347260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41485 (local address 127.5.227.129:43175)
0928 10:56:24.348214 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:24.348258 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:24.350101 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:24.351174 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:24.351188 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:24.352067 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:24.352728 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:24.352748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.354165 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.354170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:24.357280 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:24.357317 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.358402 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.358413 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:24.358794 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:24.360079 (+ 1285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:24.360120 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:24.363224 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:24.366926 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:24.366936 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:24.366957 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:24.367499 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:24.368039 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:24.368044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:24.368048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:24.368258 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:24.368853 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:24.368862 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:24.369365 (+ 503us) client_negotiation.cc:769] Sending connection context
0928 10:56:24.369684 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 10:56:24.370284 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":302,"threads_started":1}
I20260928 10:56:24.373565 25583 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41485
I20260928 10:56:24.373973 25583 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:24.375104 25583 heartbeater.cc:511] Master 127.5.227.190:41485 requested a full tablet report, sending...
I20260928 10:56:24.378358 25394 ts_manager.cc:194] Registered new tserver with Master: 12615ca5f84f432aae463a08fb08dc4c (127.5.227.129:40183)
I20260928 10:56:24.381024 25394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.129@KRBTEST.COM'} at 127.5.227.129:43175
WARNING: no policy specified for kudu/127.5.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
Entry for principal kudu/127.5.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab.
I20260928 10:56:24.450902 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41485
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:24.822436 25589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:24.822759 25589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:24.822860 25589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:24.830344 25589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:24.830476 25589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:24.830593 25589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:56:24.841945 25589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.130.keytab
--principal=kudu/127.5.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:24.843322 25589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:24.844817 25589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:24.860626 25597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:24.860611 25595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:24.861084 25594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:24.861405 25589 server_base.cc:1034] running on GCE node
I20260928 10:56:24.862076 25589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:24.863144 25589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:24.864367 25589 hybrid_clock.cc:648] HybridClock initialized: now 1790592984864324 us; error 33 us; skew 500 ppm
I20260928 10:56:24.864725 25589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:24 dist-test-slave-8w8g krb5kdc[25349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592984, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:24.868285 25589 init.cc:377] Logged in from keytab as kudu/127.5.227.130@KRBTEST.COM (short username kudu)
I20260928 10:56:24.870136 25589 webserver.cc:460] Webserver started at http://127.5.227.130:45287/ using document root <none> and password file <none>
I20260928 10:56:24.871079 25589 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:24.871239 25589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:24.871690 25589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:24.875119 25589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "33b4b630027e4c88a060af868ede6ae8"
format_stamp: "Formatted at 2026-09-28 10:56:24 on dist-test-slave-8w8g"
server_key: "56283af5b1c7f053ba8c9e42cbbad1d6"
server_key_iv: "9a06c322e80e2c8653fd3d32f2f1419e"
server_key_version: "encryptionkey@0"
I20260928 10:56:24.876077 25589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "33b4b630027e4c88a060af868ede6ae8"
format_stamp: "Formatted at 2026-09-28 10:56:24 on dist-test-slave-8w8g"
server_key: "56283af5b1c7f053ba8c9e42cbbad1d6"
server_key_iv: "9a06c322e80e2c8653fd3d32f2f1419e"
server_key_version: "encryptionkey@0"
I20260928 10:56:24.882089 25589 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:56:24.886669 25604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:24.888857 25589 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:56:24.889108 25589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "33b4b630027e4c88a060af868ede6ae8"
format_stamp: "Formatted at 2026-09-28 10:56:24 on dist-test-slave-8w8g"
server_key: "56283af5b1c7f053ba8c9e42cbbad1d6"
server_key_iv: "9a06c322e80e2c8653fd3d32f2f1419e"
server_key_version: "encryptionkey@0"
I20260928 10:56:24.889346 25589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:24.937320 25589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:24.943120 25589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:24.943465 25589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:24.944682 25589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:24.946223 25589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:24.946305 25589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:24.946450 25589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:24.946506 25589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:24.984625 25589 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:36013
I20260928 10:56:24.984721 25716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:36013 every 8 connection(s)
I20260928 10:56:24.986032 25589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:56:24.992065 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25589
I20260928 10:56:24.992260 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:56:24.992746 6030 external_mini_cluster.cc:1428] Setting key 7c0210df9bedda7990a6b468e190fbfc
Sep 28 10:56:24 dist-test-slave-8w8g krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592984, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.130@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:25.010022 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:24.989185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:35257 (local address 127.5.227.190:41485)
0928 10:56:24.989710 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:24.989716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:24.990984 (+ 1268us) server_negotiation.cc:407] Connection header received
0928 10:56:24.993239 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:24.993245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:24.993356 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:24.993495 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:24.995704 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.996534 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.998653 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.999157 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.999309 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.003222 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:25.003227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:25.003231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:25.003268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:25.006287 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.007353 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.007359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.007364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.007512 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.008101 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.008106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.008111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.008421 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:25.008602 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.009377 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.009649 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":285,"threads_started":1}
I20260928 10:56:25.011108 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:24.989777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41485 (local address 127.5.227.130:35257)
0928 10:56:24.990786 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:24.990833 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:24.992861 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:24.993964 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:24.993978 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:24.994805 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:24.995509 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:24.995529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.996725 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.996730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:24.998349 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:24.998364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:24.999431 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:24.999440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:24.999806 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.000596 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:25.000622 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:25.002915 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:25.006599 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:25.006619 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:25.006636 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:25.007192 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:25.007657 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:25.007663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:25.007667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:25.007961 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:25.008570 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:25.008580 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:25.008974 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 10:56:25.009275 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 10:56:25.009867 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":348,"threads_started":1}
I20260928 10:56:25.012780 25717 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41485
I20260928 10:56:25.013144 25717 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:25.014153 25717 heartbeater.cc:511] Master 127.5.227.190:41485 requested a full tablet report, sending...
I20260928 10:56:25.016135 25394 ts_manager.cc:194] Registered new tserver with Master: 33b4b630027e4c88a060af868ede6ae8 (127.5.227.130:36013)
I20260928 10:56:25.017410 25394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.130@KRBTEST.COM'} at 127.5.227.130:35257
WARNING: no policy specified for kudu/127.5.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
Entry for principal kudu/127.5.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab.
I20260928 10:56:25.056941 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41485
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:56:25.385421 25583 heartbeater.cc:503] Master 127.5.227.190:41485 was elected leader, sending a full tablet report...
W20260928 10:56:25.419863 25723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:25.420334 25723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:25.420542 25723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:25.429610 25723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:25.429817 25723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:25.430022 25723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:56:25.440650 25723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.131.keytab
--principal=kudu/127.5.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:25.442116 25723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:25.443710 25723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:25.460813 25731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:25.460852 25729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:25.460888 25728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:25.462165 25723 server_base.cc:1034] running on GCE node
I20260928 10:56:25.463188 25723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:25.464624 25723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:25.465960 25723 hybrid_clock.cc:648] HybridClock initialized: now 1790592985465902 us; error 60 us; skew 500 ppm
I20260928 10:56:25.466398 25723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:25 dist-test-slave-8w8g krb5kdc[25349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592985, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:25.471071 25723 init.cc:377] Logged in from keytab as kudu/127.5.227.131@KRBTEST.COM (short username kudu)
I20260928 10:56:25.473028 25723 webserver.cc:460] Webserver started at http://127.5.227.131:44065/ using document root <none> and password file <none>
I20260928 10:56:25.473831 25723 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:25.474005 25723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:25.474414 25723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:25.477953 25723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "9587a1de792f412e9bce6a997f0f21d3"
format_stamp: "Formatted at 2026-09-28 10:56:25 on dist-test-slave-8w8g"
server_key: "07360035b7ef9423ff269e6721384d08"
server_key_iv: "41d0833dfa2bdba16ed0c775c431e7cb"
server_key_version: "encryptionkey@0"
I20260928 10:56:25.478863 25723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "9587a1de792f412e9bce6a997f0f21d3"
format_stamp: "Formatted at 2026-09-28 10:56:25 on dist-test-slave-8w8g"
server_key: "07360035b7ef9423ff269e6721384d08"
server_key_iv: "41d0833dfa2bdba16ed0c775c431e7cb"
server_key_version: "encryptionkey@0"
I20260928 10:56:25.485369 25723 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 10:56:25.489912 25738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:25.491699 25723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 10:56:25.492180 25723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "9587a1de792f412e9bce6a997f0f21d3"
format_stamp: "Formatted at 2026-09-28 10:56:25 on dist-test-slave-8w8g"
server_key: "07360035b7ef9423ff269e6721384d08"
server_key_iv: "41d0833dfa2bdba16ed0c775c431e7cb"
server_key_version: "encryptionkey@0"
I20260928 10:56:25.492453 25723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:25.527436 25723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:25.533200 25723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:25.533609 25723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:25.534915 25723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:25.536900 25723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:25.537039 25723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:25.537231 25723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:25.537348 25723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:25.579360 25723 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:35909
I20260928 10:56:25.579383 25850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:35909 every 8 connection(s)
I20260928 10:56:25.581301 25723 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:56:25.587105 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25723
I20260928 10:56:25.587333 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:56:25.588100 6030 external_mini_cluster.cc:1428] Setting key 2d1c2a1f9dc5be09d50cb44d0b126722
Sep 28 10:56:25 dist-test-slave-8w8g krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592985, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.131@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:25.606130 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.584318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:52043 (local address 127.5.227.190:41485)
0928 10:56:25.584947 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.584953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.586053 (+ 1100us) server_negotiation.cc:407] Connection header received
0928 10:56:25.588655 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.588661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.588784 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.588918 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:25.590782 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.591848 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.593714 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.594152 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.594306 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.598346 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:25.598352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:25.598357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:25.598580 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:25.602027 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.603400 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.603404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.603407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.603621 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.604184 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.604189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.604192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.604606 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:25.604765 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.605590 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.605800 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":292,"threads_started":1}
I20260928 10:56:25.607085 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.585005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:41485 (local address 127.5.227.131:52043)
0928 10:56:25.585853 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:25.585904 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:25.588287 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:25.589217 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:25.589228 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:25.589855 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:25.590560 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.590584 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.592061 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.592066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.593499 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.593516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.594420 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.594428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.594718 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.595420 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:56:25.595460 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:25.598048 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:25.602304 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:25.602324 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:25.602354 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:25.603235 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:25.603835 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:56:25.603839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:56:25.603842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:56:25.604064 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:56:25.604755 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:25.604765 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:25.605157 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 10:56:25.605409 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 10:56:25.605839 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":396,"threads_started":1}
I20260928 10:56:25.608996 25851 heartbeater.cc:347] Connected to a master server at 127.5.227.190:41485
I20260928 10:56:25.609368 25851 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:25.610206 25851 heartbeater.cc:511] Master 127.5.227.190:41485 requested a full tablet report, sending...
I20260928 10:56:25.612092 25394 ts_manager.cc:194] Registered new tserver with Master: 9587a1de792f412e9bce6a997f0f21d3 (127.5.227.131:35909)
I20260928 10:56:25.612974 25394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.227.131@KRBTEST.COM'} at 127.5.227.131:52043
I20260928 10:56:25.617786 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:56:25.638788 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.622335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.5.227.190:41485)
0928 10:56:25.622559 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.622564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.623074 (+ 510us) server_negotiation.cc:407] Connection header received
0928 10:56:25.623230 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.623237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.623349 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.623470 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:25.625060 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.625810 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.627699 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.628340 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.628537 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.631326 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:25.631332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:25.631335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:25.631386 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:25.634642 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.635628 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.635633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.635635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.635890 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.636425 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.636430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.636447 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.636955 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:25.637165 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.637734 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.638303 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:56:25.644071 25394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:56:25.651702 25394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:56:25.679955 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.667342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.5.227.129:40183)
0928 10:56:25.669076 (+ 1734us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.669091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.669872 (+ 781us) server_negotiation.cc:407] Connection header received
0928 10:56:25.670357 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.670367 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.670648 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.670863 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:25.671903 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.673639 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.676730 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.678395 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.678560 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.678817 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.679341 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.679529 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":1280,"threads_started":1}
I20260928 10:56:25.681181 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.666868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40183 (local address 127.0.0.1:37722)
0928 10:56:25.669700 (+ 2832us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:25.669740 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:25.670068 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:25.671214 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:25.671222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:25.671249 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:25.671683 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.671696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.673817 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.673822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.676454 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.676477 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.678751 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.678764 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.679031 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.679096 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 10:56:25.679269 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 10:56:25.680683 (+ 1414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2578,"mutex_wait_us":22,"thread_start_us":2413,"threads_started":1}
I20260928 10:56:25.681612 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.669263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.5.227.131:35909)
0928 10:56:25.670505 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.670513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.671123 (+ 610us) server_negotiation.cc:407] Connection header received
0928 10:56:25.671290 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.671300 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.671542 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.671771 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:25.673303 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.675280 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.677862 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.679404 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.679584 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.679898 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.680612 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.681185 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":872,"threads_started":1}
I20260928 10:56:25.682228 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.667963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:35909 (local address 127.0.0.1:42968)
0928 10:56:25.670814 (+ 2851us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:25.670833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:25.670962 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:25.672124 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:25.672129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:25.672155 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:25.673045 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.673060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.675481 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.675487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.677662 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.677676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.680121 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.680131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.680336 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.680371 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:56:25.680465 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 10:56:25.681780 (+ 1315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2730,"mutex_wait_us":1146,"thread_start_us":2704,"threads_started":1}
I20260928 10:56:25.683549 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.671516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.5.227.130:36013)
0928 10:56:25.672320 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.672332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.673011 (+ 679us) server_negotiation.cc:407] Connection header received
0928 10:56:25.673128 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.673137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.673399 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.673745 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:25.674433 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.676288 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.679055 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.680797 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.680979 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.681202 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.682913 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.683188 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":507,"threads_started":1}
I20260928 10:56:25.683667 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.671026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:36013 (local address 127.0.0.1:56286)
0928 10:56:25.672775 (+ 1749us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:25.672793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:25.672922 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:25.673737 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:25.673742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:25.673771 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:25.674263 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.674276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.676434 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.676441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.678861 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.678876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.681391 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.681398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.682667 (+ 1269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.682718 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 10:56:25.682838 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 10:56:25.683007 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":9,"thread_start_us":1642,"threads_started":1}
I20260928 10:56:25.686088 25786 tablet_service.cc:1467] Processing CreateTablet for tablet 9b619717975c478d8a1e8891c4761205 (DEFAULT_TABLE table=test-table [id=e766700f18e9440faa2ccf59f19e4526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:25.686414 25518 tablet_service.cc:1467] Processing CreateTablet for tablet 9b619717975c478d8a1e8891c4761205 (DEFAULT_TABLE table=test-table [id=e766700f18e9440faa2ccf59f19e4526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:25.687631 25786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b619717975c478d8a1e8891c4761205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:25.688155 25518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b619717975c478d8a1e8891c4761205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:25.688859 25652 tablet_service.cc:1467] Processing CreateTablet for tablet 9b619717975c478d8a1e8891c4761205 (DEFAULT_TABLE table=test-table [id=e766700f18e9440faa2ccf59f19e4526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:25.690423 25652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b619717975c478d8a1e8891c4761205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:25.701782 25870 tablet_bootstrap.cc:492] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Bootstrap starting.
I20260928 10:56:25.704520 25871 tablet_bootstrap.cc:492] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Bootstrap starting.
I20260928 10:56:25.705963 25870 tablet_bootstrap.cc:654] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:25.706568 25872 tablet_bootstrap.cc:492] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Bootstrap starting.
I20260928 10:56:25.707602 25870 log.cc:824] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:25.708993 25871 tablet_bootstrap.cc:654] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:25.710321 25872 tablet_bootstrap.cc:654] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:25.710632 25871 log.cc:824] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:25.711556 25872 log.cc:824] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:25.715085 25871 tablet_bootstrap.cc:492] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: No bootstrap required, opened a new log
I20260928 10:56:25.715502 25871 ts_tablet_manager.cc:1397] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 10:56:25.715798 25870 tablet_bootstrap.cc:492] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: No bootstrap required, opened a new log
I20260928 10:56:25.715802 25872 tablet_bootstrap.cc:492] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: No bootstrap required, opened a new log
I20260928 10:56:25.716296 25872 ts_tablet_manager.cc:1397] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260928 10:56:25.716297 25870 ts_tablet_manager.cc:1397] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260928 10:56:25.724081 25871 raft_consensus.cc:348] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.724560 25871 raft_consensus.cc:374] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:25.724665 25871 raft_consensus.cc:729] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12615ca5f84f432aae463a08fb08dc4c, State: Initialized, Role: FOLLOWER
I20260928 10:56:25.725386 25871 consensus_queue.cc:260] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.728164 25871 ts_tablet_manager.cc:1428] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260928 10:56:25.728353 25870 raft_consensus.cc:348] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.728353 25872 raft_consensus.cc:348] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.728834 25872 raft_consensus.cc:374] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:25.728834 25870 raft_consensus.cc:374] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:25.728945 25870 raft_consensus.cc:729] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9587a1de792f412e9bce6a997f0f21d3, State: Initialized, Role: FOLLOWER
I20260928 10:56:25.728955 25872 raft_consensus.cc:729] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b4b630027e4c88a060af868ede6ae8, State: Initialized, Role: FOLLOWER
I20260928 10:56:25.729642 25870 consensus_queue.cc:260] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.729642 25872 consensus_queue.cc:260] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.733037 25717 heartbeater.cc:503] Master 127.5.227.190:41485 was elected leader, sending a full tablet report...
I20260928 10:56:25.733101 25851 heartbeater.cc:503] Master 127.5.227.190:41485 was elected leader, sending a full tablet report...
I20260928 10:56:25.733031 25872 ts_tablet_manager.cc:1428] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260928 10:56:25.733595 25870 ts_tablet_manager.cc:1428] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260928 10:56:25.738101 25877 raft_consensus.cc:482] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260928 10:56:25.738339 25718 tablet.cc:2369] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:25.738629 25877 raft_consensus.cc:504] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.740662 25877 leader_election.cc:288] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12615ca5f84f432aae463a08fb08dc4c (127.5.227.129:40183), 9587a1de792f412e9bce6a997f0f21d3 (127.5.227.131:35909)
I20260928 10:56:25.748346 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.740882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:40183 (local address 127.5.227.130:39441)
0928 10:56:25.742041 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:25.742075 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:25.742277 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:25.742665 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:25.742669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:25.742696 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:25.743028 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.743036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.744228 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.744237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.746091 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.746105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.747353 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.747359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.747507 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.747535 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 10:56:25.747610 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 10:56:25.747715 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":1315,"threads_started":1}
I20260928 10:56:25.748369 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.741325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:39441 (local address 127.5.227.129:40183)
0928 10:56:25.741580 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.741589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.742188 (+ 599us) server_negotiation.cc:407] Connection header received
0928 10:56:25.742430 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.742448 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.742525 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.742644 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:25.743184 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.744115 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.746265 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.747109 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.747248 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.747305 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.747890 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.748083 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:56:25.749653 25538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b619717975c478d8a1e8891c4761205" candidate_uuid: "33b4b630027e4c88a060af868ede6ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12615ca5f84f432aae463a08fb08dc4c" is_pre_election: true
I20260928 10:56:25.750268 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.741752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:58623 (local address 127.5.227.131:35909)
0928 10:56:25.741981 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.741987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.742656 (+ 669us) server_negotiation.cc:407] Connection header received
0928 10:56:25.743842 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.743848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.743925 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.744018 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:25.744714 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.745578 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.747427 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.749268 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.749465 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.749545 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.749864 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.750006 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:56:25.750312 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.741141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:35909 (local address 127.5.227.130:58623)
0928 10:56:25.742558 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:25.742577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:25.743619 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:25.744050 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:25.744055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:25.744083 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:25.744535 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.744546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.745738 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.745744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.747235 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:25.747251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.749500 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.749507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:25.749684 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.749711 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 10:56:25.749824 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 10:56:25.749971 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"thread_start_us":415,"threads_started":1}
I20260928 10:56:25.750231 25538 raft_consensus.cc:2454] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b4b630027e4c88a060af868ede6ae8 in term 0.
I20260928 10:56:25.751060 25607 leader_election.cc:302] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12615ca5f84f432aae463a08fb08dc4c, 33b4b630027e4c88a060af868ede6ae8; no voters:
I20260928 10:56:25.751322 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b619717975c478d8a1e8891c4761205" candidate_uuid: "33b4b630027e4c88a060af868ede6ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9587a1de792f412e9bce6a997f0f21d3" is_pre_election: true
I20260928 10:56:25.751549 25877 raft_consensus.cc:2781] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:56:25.751700 25877 raft_consensus.cc:482] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:56:25.751853 25877 raft_consensus.cc:3037] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:25.751972 25806 raft_consensus.cc:2454] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b4b630027e4c88a060af868ede6ae8 in term 0.
I20260928 10:56:25.754325 25877 raft_consensus.cc:504] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.755271 25877 leader_election.cc:288] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [CANDIDATE]: Term 1 election: Requested vote from peers 12615ca5f84f432aae463a08fb08dc4c (127.5.227.129:40183), 9587a1de792f412e9bce6a997f0f21d3 (127.5.227.131:35909)
I20260928 10:56:25.756029 25538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b619717975c478d8a1e8891c4761205" candidate_uuid: "33b4b630027e4c88a060af868ede6ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12615ca5f84f432aae463a08fb08dc4c"
I20260928 10:56:25.756075 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b619717975c478d8a1e8891c4761205" candidate_uuid: "33b4b630027e4c88a060af868ede6ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9587a1de792f412e9bce6a997f0f21d3"
I20260928 10:56:25.756294 25538 raft_consensus.cc:3037] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:25.756309 25806 raft_consensus.cc:3037] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:25.760387 25806 raft_consensus.cc:2454] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b4b630027e4c88a060af868ede6ae8 in term 1.
I20260928 10:56:25.760387 25538 raft_consensus.cc:2454] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b4b630027e4c88a060af868ede6ae8 in term 1.
I20260928 10:56:25.761240 25608 leader_election.cc:302] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b4b630027e4c88a060af868ede6ae8, 9587a1de792f412e9bce6a997f0f21d3; no voters:
I20260928 10:56:25.761775 25877 raft_consensus.cc:2781] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:25.762342 25877 raft_consensus.cc:686] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 33b4b630027e4c88a060af868ede6ae8, State: Running, Role: LEADER
I20260928 10:56:25.763103 25877 consensus_queue.cc:237] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } }
I20260928 10:56:25.768402 25392 catalog_manager.cc:5495] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33b4b630027e4c88a060af868ede6ae8 (127.5.227.130). New cstate: current_term: 1 leader_uuid: "33b4b630027e4c88a060af868ede6ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33b4b630027e4c88a060af868ede6ae8" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36013 } health_report { overall_health: HEALTHY } } }
I20260928 10:56:25.814018 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.806420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.5.227.130:36013)
0928 10:56:25.806652 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.806657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.806931 (+ 274us) server_negotiation.cc:407] Connection header received
0928 10:56:25.807145 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.807150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.807237 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.807411 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:56:25.808140 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.809074 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.811083 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.811546 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.811666 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.812565 (+ 899us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:56:25.813103 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:56:25.813244 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.813505 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.813590 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:56:25.823477 25538 raft_consensus.cc:1261] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 1 FOLLOWER]: Refusing update from remote peer 33b4b630027e4c88a060af868ede6ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:25.823475 25806 raft_consensus.cc:1261] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 1 FOLLOWER]: Refusing update from remote peer 33b4b630027e4c88a060af868ede6ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:25.824671 25881 consensus_queue.cc:1035] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12615ca5f84f432aae463a08fb08dc4c" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:25.825138 25877 consensus_queue.cc:1035] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9587a1de792f412e9bce6a997f0f21d3" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:56:25.833248 25852 tablet.cc:2369] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:25.838213 25884 mvcc.cc:205] Tried to move back new op lower bound from 7334268869921701888 to 7334268869690957824. Current Snapshot: MvccSnapshot[applied={T|T < 7334268869921701888}]
I20260928 10:56:25.856449 25392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:25.856791 25392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:25.861646 25392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 748e83475f45448abcddd58e9fc52a48: Sending DeleteTablet for 3 replicas of tablet 9b619717975c478d8a1e8891c4761205
I20260928 10:56:25.863076 25518 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b619717975c478d8a1e8891c4761205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:25 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:37722
I20260928 10:56:25.863255 25652 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b619717975c478d8a1e8891c4761205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:25 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:56286
I20260928 10:56:25.863329 25786 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b619717975c478d8a1e8891c4761205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:25 UTC) from {username='kudu', principal='kudu/127.5.227.190@KRBTEST.COM'} at 127.0.0.1:42968
I20260928 10:56:25.865006 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25455
I20260928 10:56:25.875929 25894 tablet_replica.cc:330] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: stopping tablet replica
I20260928 10:56:25.876811 25894 raft_consensus.cc:2229] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:25.877671 25894 raft_consensus.cc:2258] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:25.878749 25895 tablet_replica.cc:330] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: stopping tablet replica
I20260928 10:56:25.878991 25896 tablet_replica.cc:330] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: stopping tablet replica
I20260928 10:56:25.879537 25896 raft_consensus.cc:2229] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:25.880196 25896 raft_consensus.cc:2258] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:25.881819 25895 raft_consensus.cc:2229] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:56:25.882090 25894 ts_tablet_manager.cc:1905] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:25.884498 25896 ts_tablet_manager.cc:1905] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:25.886930 25895 raft_consensus.cc:2258] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:25.890789 25894 ts_tablet_manager.cc:1918] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 10:56:25 dist-test-slave-8w8g krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.129@KRBTEST.COM
I20260928 10:56:25.891136 25894 log.cc:1196] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-0/wal/wals/9b619717975c478d8a1e8891c4761205
I20260928 10:56:25.891871 25894 ts_tablet_manager.cc:1939] T 9b619717975c478d8a1e8891c4761205 P 12615ca5f84f432aae463a08fb08dc4c: Deleting consensus metadata
I20260928 10:56:25.892773 25896 ts_tablet_manager.cc:1918] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:25.892968 25896 log.cc:1196] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/9b619717975c478d8a1e8891c4761205
I20260928 10:56:25.893440 25895 ts_tablet_manager.cc:1905] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:25.893611 25896 ts_tablet_manager.cc:1939] T 9b619717975c478d8a1e8891c4761205 P 9587a1de792f412e9bce6a997f0f21d3: Deleting consensus metadata
I20260928 10:56:25.894395 25380 catalog_manager.cc:4841] TS 12615ca5f84f432aae463a08fb08dc4c (127.5.227.129:40183): tablet 9b619717975c478d8a1e8891c4761205 (table test-table [id=e766700f18e9440faa2ccf59f19e4526]) successfully deleted
I20260928 10:56:25.895605 25381 catalog_manager.cc:4841] TS 9587a1de792f412e9bce6a997f0f21d3 (127.5.227.131:35909): tablet 9b619717975c478d8a1e8891c4761205 (table test-table [id=e766700f18e9440faa2ccf59f19e4526]) successfully deleted
I20260928 10:56:25.902686 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:25.866498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.5.227.129:40183)
0928 10:56:25.866696 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:25.866702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:25.872764 (+ 6062us) server_negotiation.cc:407] Connection header received
0928 10:56:25.873018 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:25.873061 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:25.873171 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:25.873299 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:25.881043 (+ 7744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.882363 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.884921 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:25.885696 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:25.885820 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:25.895697 (+ 9877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:25.895715 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:25.895764 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:25.895817 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:25.899306 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.900334 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.900343 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.900346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.900512 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:25.900973 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:25.900983 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:25.900986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:25.901376 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:25.901725 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:25.902058 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 10:56:25.902300 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:56:25.903136 25895 ts_tablet_manager.cc:1918] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:25.903337 25895 log.cc:1196] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/9b619717975c478d8a1e8891c4761205
I20260928 10:56:25.903450 25578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:25.903962 25895 ts_tablet_manager.cc:1939] T 9b619717975c478d8a1e8891c4761205 P 33b4b630027e4c88a060af868ede6ae8: Deleting consensus metadata
I20260928 10:56:25.906530 25381 catalog_manager.cc:4841] TS 33b4b630027e4c88a060af868ede6ae8 (127.5.227.130:36013): tablet 9b619717975c478d8a1e8891c4761205 (table test-table [id=e766700f18e9440faa2ccf59f19e4526]) successfully deleted
I20260928 10:56:26.030865 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25455
I20260928 10:56:26.049587 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25589
Sep 28 10:56:26 dist-test-slave-8w8g krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.130@KRBTEST.COM
I20260928 10:56:26.072088 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:26.050471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.5.227.130:36013)
0928 10:56:26.050674 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:26.050679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:26.050700 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:56:26.051216 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:26.051221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:26.051297 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:26.051420 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:26.053411 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:26.054665 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:26.056251 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:26.056819 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:26.056988 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:26.062725 (+ 5737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:26.062748 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:26.062773 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:26.062847 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:26.067030 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:26.068617 (+ 1587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:26.068626 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:26.068629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:26.068853 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:26.069613 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:26.069625 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:26.069627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:26.070072 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:26.070588 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:26.070807 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 10:56:26.071228 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 10:56:26.073078 25712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:26.201992 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25589
I20260928 10:56:26.221578 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25723
Sep 28 10:56:26 dist-test-slave-8w8g krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.131@KRBTEST.COM
I20260928 10:56:26.242058 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:26.222905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.5.227.131:35909)
0928 10:56:26.223173 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:26.223178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:26.223197 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:56:26.223257 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:26.223261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:26.223326 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:26.223445 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:26.225423 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:26.226294 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:26.227891 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:26.228359 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:26.228480 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:26.234185 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:26.234202 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:26.234218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:26.234266 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:26.238252 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:26.239214 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:26.239223 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:26.239226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:26.239370 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:26.240050 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:26.240055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:26.240058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:26.240505 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:26.240891 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:26.241219 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 10:56:26.241606 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 10:56:26.243027 25846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:26.344803 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25723
I20260928 10:56:26.364648 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25362
I20260928 10:56:26.366400 25424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:26.494865 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25362
2026-09-28T10:56:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4891 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T10:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:26Z Disabled control of system clock
I20260928 10:56:26.580919 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37055
--webserver_interface=127.5.227.190
--webserver_port=0
--builtin_ntp_servers=127.5.227.148:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:56:26.964098 25905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:26.964426 25905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:26.964561 25905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:26.972538 25905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:26.972681 25905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:26.972721 25905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:26.972818 25905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:26.972895 25905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:26.983806 25905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:37055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:26.985553 25905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:26.987756 25905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:27.002351 25910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:27.002424 25911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:27.002652 25913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:27.004531 25905 server_base.cc:1034] running on GCE node
I20260928 10:56:27.005541 25905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:27.007692 25905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:27.009102 25905 hybrid_clock.cc:648] HybridClock initialized: now 1790592987009021 us; error 47 us; skew 500 ppm
I20260928 10:56:27.009690 25905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:27.014402 25905 webserver.cc:460] Webserver started at http://127.5.227.190:39271/ using document root <none> and password file <none>
I20260928 10:56:27.016016 25905 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:27.016192 25905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:27.016563 25905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:27.020354 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "bf00a05d12c1419da008efae4691a63b"
format_stamp: "Formatted at 2026-09-28 10:56:27 on dist-test-slave-8w8g"
server_key: "8f493fea0f32b506ac429b8f0f4ef407"
server_key_iv: "b61319032d64c4f3fa26f433b6f2e4fe"
server_key_version: "encryptionkey@0"
I20260928 10:56:27.021342 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "bf00a05d12c1419da008efae4691a63b"
format_stamp: "Formatted at 2026-09-28 10:56:27 on dist-test-slave-8w8g"
server_key: "8f493fea0f32b506ac429b8f0f4ef407"
server_key_iv: "b61319032d64c4f3fa26f433b6f2e4fe"
server_key_version: "encryptionkey@0"
I20260928 10:56:27.028296 25905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 10:56:27.034660 25919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:27.036649 25905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:27.036916 25905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "bf00a05d12c1419da008efae4691a63b"
format_stamp: "Formatted at 2026-09-28 10:56:27 on dist-test-slave-8w8g"
server_key: "8f493fea0f32b506ac429b8f0f4ef407"
server_key_iv: "b61319032d64c4f3fa26f433b6f2e4fe"
server_key_version: "encryptionkey@0"
I20260928 10:56:27.037237 25905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:27.066725 25905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:27.067855 25905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:27.068184 25905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:27.093508 25905 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:37055
I20260928 10:56:27.093530 25970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:37055 every 8 connection(s)
I20260928 10:56:27.095211 25905 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:27.100284 25971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:27.101594 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25905
I20260928 10:56:27.101795 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:27.102332 6030 external_mini_cluster.cc:1428] Setting key a56315c025189f2c8668b1a52564de2d
I20260928 10:56:27.118566 25971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b: Bootstrap starting.
I20260928 10:56:27.121611 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:27.105158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.5.227.190:37055)
0928 10:56:27.106615 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:27.106631 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:27.106693 (+ 62us) server_negotiation.cc:407] Connection header received
0928 10:56:27.108710 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:27.108743 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:27.109392 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:27.109808 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:27.110650 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:27.112534 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:27.114757 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:27.116314 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:27.116544 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:27.118601 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:27.118616 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:27.118634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:27.119315 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:27.119403 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:27.119431 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:27.119529 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:27.120159 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 10:56:27.120532 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":494,"threads_started":1}
I20260928 10:56:27.122435 25971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:27.123791 25971 log.cc:824] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:27.127364 25971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b: No bootstrap required, opened a new log
I20260928 10:56:27.137801 25971 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf00a05d12c1419da008efae4691a63b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37055 } }
I20260928 10:56:27.138320 25971 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:27.138453 25971 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf00a05d12c1419da008efae4691a63b, State: Initialized, Role: FOLLOWER
I20260928 10:56:27.139302 25971 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf00a05d12c1419da008efae4691a63b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37055 } }
I20260928 10:56:27.139674 25971 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:27.139926 25971 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:27.140143 25971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:27.142972 25971 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf00a05d12c1419da008efae4691a63b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37055 } }
I20260928 10:56:27.143652 25971 leader_election.cc:302] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf00a05d12c1419da008efae4691a63b; no voters:
I20260928 10:56:27.144639 25971 leader_election.cc:288] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:27.144755 25976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:27.146201 25976 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [term 1 LEADER]: Becoming Leader. State: Replica: bf00a05d12c1419da008efae4691a63b, State: Running, Role: LEADER
I20260928 10:56:27.146876 25976 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf00a05d12c1419da008efae4691a63b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37055 } }
I20260928 10:56:27.147639 25971 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:27.150323 25977 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf00a05d12c1419da008efae4691a63b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf00a05d12c1419da008efae4691a63b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37055 } } }
I20260928 10:56:27.151069 25977 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:27.151801 25978 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf00a05d12c1419da008efae4691a63b. Latest consensus state: current_term: 1 leader_uuid: "bf00a05d12c1419da008efae4691a63b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf00a05d12c1419da008efae4691a63b" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37055 } } }
I20260928 10:56:27.152148 25978 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:27.153827 25982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:27.160122 25982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:27.176085 25982 catalog_manager.cc:1347] Generated new cluster ID: b06e3ad03f4446c9aecd9c59d55f0b45
I20260928 10:56:27.176280 25982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:27.193866 25982 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:27.196247 25982 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:27.208997 25982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b: Generated new TSK 0
I20260928 10:56:27.210156 25982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:27.229964 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.129:0
--local_ip_for_outbound_sockets=127.5.227.129
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37055
--builtin_ntp_servers=127.5.227.148:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:56:27.608014 25995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:27.608319 25995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:27.608474 25995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:27.616341 25995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:27.616513 25995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:27.616686 25995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.129
I20260928 10:56:27.628139 25995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.227.129
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.129
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:27.629756 25995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:27.631721 25995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:27.648110 26003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:27.648249 26000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:27.648226 26001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:27.649225 25995 server_base.cc:1034] running on GCE node
I20260928 10:56:27.650103 25995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:27.651182 25995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:27.652428 25995 hybrid_clock.cc:648] HybridClock initialized: now 1790592987652344 us; error 71 us; skew 500 ppm
I20260928 10:56:27.653167 25995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:27.656174 25995 webserver.cc:460] Webserver started at http://127.5.227.129:42769/ using document root <none> and password file <none>
I20260928 10:56:27.657027 25995 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:27.657155 25995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:27.657583 25995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:27.661167 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data/instance:
uuid: "8b596f449af84e9cb30c84bf18f0242d"
format_stamp: "Formatted at 2026-09-28 10:56:27 on dist-test-slave-8w8g"
server_key: "5656273bcb20852260a4fb01349e9c43"
server_key_iv: "2df5eb0eebbb442bd69b27c147f1f073"
server_key_version: "encryptionkey@0"
I20260928 10:56:27.662149 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance:
uuid: "8b596f449af84e9cb30c84bf18f0242d"
format_stamp: "Formatted at 2026-09-28 10:56:27 on dist-test-slave-8w8g"
server_key: "5656273bcb20852260a4fb01349e9c43"
server_key_iv: "2df5eb0eebbb442bd69b27c147f1f073"
server_key_version: "encryptionkey@0"
I20260928 10:56:27.668529 25995 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 10:56:27.672886 26009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:27.674979 25995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:56:27.675172 25995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/wal
uuid: "8b596f449af84e9cb30c84bf18f0242d"
format_stamp: "Formatted at 2026-09-28 10:56:27 on dist-test-slave-8w8g"
server_key: "5656273bcb20852260a4fb01349e9c43"
server_key_iv: "2df5eb0eebbb442bd69b27c147f1f073"
server_key_version: "encryptionkey@0"
I20260928 10:56:27.675493 25995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:27.706306 25995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:27.707371 25995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:27.707849 25995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:27.709076 25995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:27.711061 25995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:27.711225 25995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:27.711382 25995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:27.711488 25995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:27.758276 25995 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.129:42099
I20260928 10:56:27.758340 26121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.129:42099 every 8 connection(s)
I20260928 10:56:27.760036 25995 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/data/info.pb
I20260928 10:56:27.770208 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 25995
I20260928 10:56:27.770406 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-0/wal/instance
I20260928 10:56:27.770942 6030 external_mini_cluster.cc:1428] Setting key 7c7c0d11e10aaf084a8ed12b1eb4b669
I20260928 10:56:27.774480 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.130:0
--local_ip_for_outbound_sockets=127.5.227.130
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37055
--builtin_ntp_servers=127.5.227.148:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:56:27.775772 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:27.763268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:35007 (local address 127.5.227.190:37055)
0928 10:56:27.763896 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:27.763902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:27.765173 (+ 1271us) server_negotiation.cc:407] Connection header received
0928 10:56:27.767150 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:27.767157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:27.767254 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:27.767380 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:27.768645 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:27.769378 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:27.771421 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:27.772000 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:27.772141 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:27.774250 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:27.774258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:27.774263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:27.774289 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:27.774349 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:27.774367 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:27.774488 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:27.775052 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 10:56:27.775192 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":340,"threads_started":1}
I20260928 10:56:27.777041 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:27.764048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37055 (local address 127.5.227.129:35007)
0928 10:56:27.764974 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:27.765042 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:27.766811 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:27.767640 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:27.767649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:27.767822 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:27.768448 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:27.768474 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:27.769636 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:27.769642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:27.771241 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:27.771258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:27.772338 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:27.772356 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:27.772804 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:27.773846 (+ 1042us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:27.773849 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:27.773896 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:27.774005 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:27.774678 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:27.774735 (+ 57us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:27.774746 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:56:27.774975 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 10:56:27.775396 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":264,"threads_started":1}
I20260928 10:56:27.779326 26122 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37055
I20260928 10:56:27.779773 26122 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:27.781016 26122 heartbeater.cc:511] Master 127.5.227.190:37055 requested a full tablet report, sending...
I20260928 10:56:27.784454 25936 ts_manager.cc:194] Registered new tserver with Master: 8b596f449af84e9cb30c84bf18f0242d (127.5.227.129:42099)
I20260928 10:56:27.786190 25936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.129:35007
W20260928 10:56:28.167872 26126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:28.168248 26126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:28.168687 26126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:28.180729 26126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:28.180940 26126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:28.181182 26126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.130
I20260928 10:56:28.193558 26126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.227.130
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.130
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:28.195250 26126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:28.198436 26126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:28.217449 26131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:28.217628 26134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:28.217657 26132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:28.220352 26126 server_base.cc:1034] running on GCE node
I20260928 10:56:28.221446 26126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:28.222741 26126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:28.224064 26126 hybrid_clock.cc:648] HybridClock initialized: now 1790592988224047 us; error 66 us; skew 500 ppm
I20260928 10:56:28.224534 26126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:28.228259 26126 webserver.cc:460] Webserver started at http://127.5.227.130:33845/ using document root <none> and password file <none>
I20260928 10:56:28.229152 26126 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:28.229321 26126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:28.229691 26126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:28.235401 26126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data/instance:
uuid: "d9d24f3b44984ceba67cf5c266287467"
format_stamp: "Formatted at 2026-09-28 10:56:28 on dist-test-slave-8w8g"
server_key: "98cfaf95ad2b418926109a19b5dfe478"
server_key_iv: "349c694ce75d28ad400edd1111728a0f"
server_key_version: "encryptionkey@0"
I20260928 10:56:28.236756 26126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance:
uuid: "d9d24f3b44984ceba67cf5c266287467"
format_stamp: "Formatted at 2026-09-28 10:56:28 on dist-test-slave-8w8g"
server_key: "98cfaf95ad2b418926109a19b5dfe478"
server_key_iv: "349c694ce75d28ad400edd1111728a0f"
server_key_version: "encryptionkey@0"
I20260928 10:56:28.243978 26126 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 10:56:28.249449 26140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:28.251901 26126 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:56:28.252187 26126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal
uuid: "d9d24f3b44984ceba67cf5c266287467"
format_stamp: "Formatted at 2026-09-28 10:56:28 on dist-test-slave-8w8g"
server_key: "98cfaf95ad2b418926109a19b5dfe478"
server_key_iv: "349c694ce75d28ad400edd1111728a0f"
server_key_version: "encryptionkey@0"
I20260928 10:56:28.252470 26126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:28.284157 26126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:28.285158 26126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:28.285526 26126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:28.286882 26126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:28.288782 26126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:28.288919 26126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:28.289036 26126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:28.289114 26126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:28.337085 26126 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.130:36473
I20260928 10:56:28.337298 26252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.130:36473 every 8 connection(s)
I20260928 10:56:28.338547 26126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/data/info.pb
I20260928 10:56:28.340234 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 26126
I20260928 10:56:28.340430 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal/instance
I20260928 10:56:28.341172 6030 external_mini_cluster.cc:1428] Setting key b2e585bf87016ba30c3ab0339ff5ce52
I20260928 10:56:28.345827 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.227.131:0
--local_ip_for_outbound_sockets=127.5.227.131
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37055
--builtin_ntp_servers=127.5.227.148:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:56:28.359431 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.342786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.130:58225 (local address 127.5.227.190:37055)
0928 10:56:28.344140 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:28.344150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:28.345361 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 10:56:28.348271 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:28.348277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:28.348379 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:28.348748 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:28.350234 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.351427 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.354781 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.355860 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.356041 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.357772 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:28.357781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:28.357789 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:28.357820 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:28.357876 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:28.357895 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:28.358007 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:28.358575 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 10:56:28.358721 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":588,"threads_started":1}
I20260928 10:56:28.360539 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.343634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37055 (local address 127.5.227.130:58225)
0928 10:56:28.345188 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:28.345236 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:28.347900 (+ 2664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:28.348986 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:28.349001 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:28.349135 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:28.349999 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.350023 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.351785 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.351793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.354341 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.354366 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.356177 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.356184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.356446 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.357295 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:28.357298 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:28.357345 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:28.357469 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:28.358102 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:28.358179 (+ 77us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:28.358191 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:56:28.358486 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 10:56:28.359054 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":559,"threads_started":1}
I20260928 10:56:28.362955 26253 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37055
I20260928 10:56:28.363997 26253 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:28.365212 26253 heartbeater.cc:511] Master 127.5.227.190:37055 requested a full tablet report, sending...
I20260928 10:56:28.367609 25936 ts_manager.cc:194] Registered new tserver with Master: d9d24f3b44984ceba67cf5c266287467 (127.5.227.130:36473)
I20260928 10:56:28.368683 25936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.130:58225
W20260928 10:56:28.759845 26257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:28.760216 26257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:28.760401 26257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:28.769461 26257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:28.769659 26257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:28.769869 26257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.227.131
I20260928 10:56:28.780612 26257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.227.131
--webserver_port=0
--tserver_master_addrs=127.5.227.190:37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.227.131
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:28.782141 26257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:28.784092 26257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 10:56:28.790246 26122 heartbeater.cc:503] Master 127.5.227.190:37055 was elected leader, sending a full tablet report...
W20260928 10:56:28.800454 26263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:28.800443 26265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:28.800894 26262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:28.801443 26257 server_base.cc:1034] running on GCE node
I20260928 10:56:28.802156 26257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:28.803414 26257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:28.805002 26257 hybrid_clock.cc:648] HybridClock initialized: now 1790592988804866 us; error 71 us; skew 500 ppm
I20260928 10:56:28.805425 26257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:56:28.815356 26257 webserver.cc:460] Webserver started at http://127.5.227.131:46183/ using document root <none> and password file <none>
I20260928 10:56:28.816301 26257 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:28.816442 26257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:28.816975 26257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:28.820323 26257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data/instance:
uuid: "b3dce68ea6c1420d9819765b021c186b"
format_stamp: "Formatted at 2026-09-28 10:56:28 on dist-test-slave-8w8g"
server_key: "d8af10579e0b1a18704d71d89f4dec92"
server_key_iv: "2f6d091da9f860560671038844e05226"
server_key_version: "encryptionkey@0"
I20260928 10:56:28.821224 26257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance:
uuid: "b3dce68ea6c1420d9819765b021c186b"
format_stamp: "Formatted at 2026-09-28 10:56:28 on dist-test-slave-8w8g"
server_key: "d8af10579e0b1a18704d71d89f4dec92"
server_key_iv: "2f6d091da9f860560671038844e05226"
server_key_version: "encryptionkey@0"
I20260928 10:56:28.826972 26257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:56:28.832484 26271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:28.834317 26257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:56:28.834757 26257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal
uuid: "b3dce68ea6c1420d9819765b021c186b"
format_stamp: "Formatted at 2026-09-28 10:56:28 on dist-test-slave-8w8g"
server_key: "d8af10579e0b1a18704d71d89f4dec92"
server_key_iv: "2f6d091da9f860560671038844e05226"
server_key_version: "encryptionkey@0"
I20260928 10:56:28.835019 26257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:28.856616 26257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:28.857555 26257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:28.857936 26257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:28.859236 26257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:56:28.861253 26257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:56:28.861356 26257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:28.861482 26257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:56:28.861557 26257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:28.902617 26257 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.131:45569
I20260928 10:56:28.902673 26383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.131:45569 every 8 connection(s)
I20260928 10:56:28.904109 26257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/data/info.pb
I20260928 10:56:28.912040 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 26257
I20260928 10:56:28.912257 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal/instance
I20260928 10:56:28.913103 6030 external_mini_cluster.cc:1428] Setting key f2853a7db42130325a675bf2b567c6b8
I20260928 10:56:28.919859 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.906994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.131:51239 (local address 127.5.227.190:37055)
0928 10:56:28.907427 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:28.907438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:28.908777 (+ 1339us) server_negotiation.cc:407] Connection header received
0928 10:56:28.910702 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:28.910708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:28.910809 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:28.910936 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:28.912207 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.913210 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.916040 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.916758 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.917061 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.918819 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:28.918825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:28.918829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:28.918850 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:28.918894 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:28.918908 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:28.918983 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:28.919357 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 10:56:28.919467 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":219,"threads_started":1}
I20260928 10:56:28.920992 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.907655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:37055 (local address 127.5.227.131:51239)
0928 10:56:28.908600 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:28.908645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:28.910387 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:28.911208 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:28.911217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:28.911318 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:56:28.912018 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.912038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.913501 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.913508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.915805 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.915828 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.917276 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.917285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.917564 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.918366 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:56:28.918369 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:56:28.918417 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:56:28.918519 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:56:28.919033 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:56:28.919044 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:56:28.919052 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:56:28.919260 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 10:56:28.919683 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":287,"threads_started":1}
I20260928 10:56:28.923048 26384 heartbeater.cc:347] Connected to a master server at 127.5.227.190:37055
I20260928 10:56:28.923549 26384 heartbeater.cc:464] Registering TS with master...
I20260928 10:56:28.925122 26384 heartbeater.cc:511] Master 127.5.227.190:37055 requested a full tablet report, sending...
I20260928 10:56:28.926913 25936 ts_manager.cc:194] Registered new tserver with Master: b3dce68ea6c1420d9819765b021c186b (127.5.227.131:45569)
I20260928 10:56:28.928332 25936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.227.131:51239
I20260928 10:56:28.930753 6030 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:56:28.942912 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.935599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.5.227.190:37055)
0928 10:56:28.935913 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:28.935920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:28.936119 (+ 199us) server_negotiation.cc:407] Connection header received
0928 10:56:28.936294 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:28.936299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:28.936379 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:28.936522 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:28.937090 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.937842 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.939425 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.940067 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.940237 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.941771 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:28.941776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:28.941780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:28.941802 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:28.941838 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:28.941851 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:28.941930 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:28.942456 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 10:56:28.942564 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 10:56:28.949031 25936 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:56:28.957464 25936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:56:28.984072 26397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.971842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:45569 (local address 127.0.0.1:47666)
0928 10:56:28.972752 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:28.972811 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:28.973026 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:28.975110 (+ 2084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:28.975117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:28.975144 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:28.975619 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.975633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.977632 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.977641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.980724 (+ 3083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.980739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.982974 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.982979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.983189 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.983254 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 10:56:28.983383 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 10:56:28.983610 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":405,"thread_start_us":967,"threads_started":1}
I20260928 10:56:28.984438 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.972119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.5.227.130:36473)
0928 10:56:28.973660 (+ 1541us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:28.973673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:28.973711 (+ 38us) server_negotiation.cc:407] Connection header received
0928 10:56:28.973818 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:28.973825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:28.974137 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:28.974955 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:28.975240 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.976933 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.981245 (+ 4312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.982640 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.982847 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.983180 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:28.983850 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 10:56:28.984046 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1328,"thread_start_us":1278,"threads_started":1}
I20260928 10:56:28.984728 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.974285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.5.227.129:42099)
0928 10:56:28.974922 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:28.974932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:28.974960 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:56:28.975052 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:28.975059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:28.975284 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:28.976125 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:28.976327 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.977758 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.980974 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.982684 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.983136 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.983429 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:28.984173 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 10:56:28.984356 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":399,"threads_started":1}
I20260928 10:56:28.985189 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.972414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:36473 (local address 127.0.0.1:39706)
0928 10:56:28.973180 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:28.973197 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:28.973315 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:28.974382 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:28.974390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:28.974425 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:28.974857 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.974872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.977101 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.977107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.980986 (+ 3879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.980998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.982904 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.982911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.983532 (+ 621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.983636 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 10:56:28.984652 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0928 10:56:28.984812 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":903,"thread_start_us":423,"threads_started":1}
I20260928 10:56:28.985278 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.973675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:43978)
0928 10:56:28.974263 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:28.974281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:28.974382 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:28.975515 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:28.975519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:28.975542 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:28.976008 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.976021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.977923 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.978014 (+ 91us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.980714 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:28.980731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.982960 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.982968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:28.983578 (+ 610us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.983652 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 10:56:28.983778 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 10:56:28.983983 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":258,"thread_start_us":819,"threads_started":1}
I20260928 10:56:28.986474 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:28.972529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.5.227.131:45569)
0928 10:56:28.974083 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:28.974094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:28.974130 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:56:28.974230 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:28.974237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:28.974463 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:28.975916 (+ 1453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:28.976198 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.977491 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.981076 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:28.982684 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:28.983513 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:28.985903 (+ 2390us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:28.985984 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 10:56:28.986154 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":1196,"threads_started":1}
I20260928 10:56:28.990506 26188 tablet_service.cc:1467] Processing CreateTablet for tablet 800630b22322449c866617388bb987fd (DEFAULT_TABLE table=test-table [id=292a209663e0450b95b2115f7c5e0ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:28.990536 26057 tablet_service.cc:1467] Processing CreateTablet for tablet 800630b22322449c866617388bb987fd (DEFAULT_TABLE table=test-table [id=292a209663e0450b95b2115f7c5e0ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:28.990661 26319 tablet_service.cc:1467] Processing CreateTablet for tablet 800630b22322449c866617388bb987fd (DEFAULT_TABLE table=test-table [id=292a209663e0450b95b2115f7c5e0ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:56:28.992157 26188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800630b22322449c866617388bb987fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:28.992156 26057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800630b22322449c866617388bb987fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:28.992182 26319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800630b22322449c866617388bb987fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:29.005970 26403 tablet_bootstrap.cc:492] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Bootstrap starting.
I20260928 10:56:29.007943 26404 tablet_bootstrap.cc:492] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: Bootstrap starting.
I20260928 10:56:29.009761 26405 tablet_bootstrap.cc:492] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Bootstrap starting.
I20260928 10:56:29.010188 26403 tablet_bootstrap.cc:654] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:29.011816 26403 log.cc:824] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:29.011904 26404 tablet_bootstrap.cc:654] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:29.012988 26405 tablet_bootstrap.cc:654] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:29.013291 26404 log.cc:824] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:29.014263 26405 log.cc:824] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:29.016152 26403 tablet_bootstrap.cc:492] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: No bootstrap required, opened a new log
I20260928 10:56:29.016506 26403 ts_tablet_manager.cc:1397] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260928 10:56:29.017282 26404 tablet_bootstrap.cc:492] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: No bootstrap required, opened a new log
I20260928 10:56:29.017591 26404 ts_tablet_manager.cc:1397] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 10:56:29.017748 26405 tablet_bootstrap.cc:492] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: No bootstrap required, opened a new log
I20260928 10:56:29.018069 26405 ts_tablet_manager.cc:1397] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 10:56:29.026818 26405 raft_consensus.cc:348] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.027330 26405 raft_consensus.cc:374] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:29.027439 26405 raft_consensus.cc:729] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d24f3b44984ceba67cf5c266287467, State: Initialized, Role: FOLLOWER
I20260928 10:56:29.028038 26403 raft_consensus.cc:348] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.028198 26405 consensus_queue.cc:260] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.028498 26403 raft_consensus.cc:374] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:29.028596 26403 raft_consensus.cc:729] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dce68ea6c1420d9819765b021c186b, State: Initialized, Role: FOLLOWER
I20260928 10:56:29.028591 26404 raft_consensus.cc:348] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.029040 26404 raft_consensus.cc:374] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:29.029140 26404 raft_consensus.cc:729] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b596f449af84e9cb30c84bf18f0242d, State: Initialized, Role: FOLLOWER
I20260928 10:56:29.029744 26253 heartbeater.cc:503] Master 127.5.227.190:37055 was elected leader, sending a full tablet report...
I20260928 10:56:29.029824 26404 consensus_queue.cc:260] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.029404 26403 consensus_queue.cc:260] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.030367 26405 ts_tablet_manager.cc:1428] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260928 10:56:29.031960 26404 ts_tablet_manager.cc:1428] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260928 10:56:29.033241 26384 heartbeater.cc:503] Master 127.5.227.190:37055 was elected leader, sending a full tablet report...
I20260928 10:56:29.034119 26403 ts_tablet_manager.cc:1428] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
W20260928 10:56:29.091400 26254 tablet.cc:2369] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:56:29.156023 26385 tablet.cc:2369] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:29.184847 26410 raft_consensus.cc:482] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:56:29.185236 26410 raft_consensus.cc:504] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.186995 26410 leader_election.cc:288] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d24f3b44984ceba67cf5c266287467 (127.5.227.130:36473), b3dce68ea6c1420d9819765b021c186b (127.5.227.131:45569)
I20260928 10:56:29.198403 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.187881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:50081 (local address 127.5.227.131:45569)
0928 10:56:29.188131 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:29.188137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:29.188776 (+ 639us) server_negotiation.cc:407] Connection header received
0928 10:56:29.188953 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:29.188959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:29.189102 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:29.189207 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:29.190613 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.191767 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.195545 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.196902 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.197115 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.197195 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:29.197954 (+ 759us) server_negotiation.cc:299] Negotiation successful
0928 10:56:29.198110 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 10:56:29.198402 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.187813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.227.129:34665 (local address 127.5.227.130:36473)
0928 10:56:29.188054 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:29.188061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:29.188777 (+ 716us) server_negotiation.cc:407] Connection header received
0928 10:56:29.188913 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:29.188919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:29.189030 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:29.189172 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:56:29.190613 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.191767 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.195503 (+ 3736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.196901 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.197115 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.197192 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:29.197996 (+ 804us) server_negotiation.cc:299] Negotiation successful
0928 10:56:29.198137 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:56:29.198544 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.187587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.131:45569 (local address 127.5.227.129:50081)
0928 10:56:29.188537 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.188578 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:29.188807 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:29.189392 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:29.189398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:29.189547 (+ 149us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:29.190274 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:29.190287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.191939 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.191953 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:29.195145 (+ 3192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:29.195165 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.197268 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.197278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:29.197557 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.197662 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 10:56:29.197786 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 10:56:29.198004 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":427,"spinlock_wait_cycles":60288,"thread_start_us":656,"threads_started":1}
I20260928 10:56:29.198500 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.187667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.130:36473 (local address 127.5.227.129:34665)
0928 10:56:29.188624 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.188645 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:56:29.188763 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:56:29.189392 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:56:29.189398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:56:29.189547 (+ 149us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:56:29.190233 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:29.190251 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.191939 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.191949 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:29.195140 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:56:29.195165 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.197285 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.197292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:56:29.197593 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.197660 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 10:56:29.197786 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 10:56:29.198004 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":501,"spinlock_wait_cycles":18048,"thread_start_us":439,"threads_started":1}
I20260928 10:56:29.200096 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800630b22322449c866617388bb987fd" candidate_uuid: "8b596f449af84e9cb30c84bf18f0242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dce68ea6c1420d9819765b021c186b" is_pre_election: true
I20260928 10:56:29.200546 26208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800630b22322449c866617388bb987fd" candidate_uuid: "8b596f449af84e9cb30c84bf18f0242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d24f3b44984ceba67cf5c266287467" is_pre_election: true
I20260928 10:56:29.200733 26339 raft_consensus.cc:2454] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b596f449af84e9cb30c84bf18f0242d in term 0.
I20260928 10:56:29.201076 26208 raft_consensus.cc:2454] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b596f449af84e9cb30c84bf18f0242d in term 0.
I20260928 10:56:29.201669 26011 leader_election.cc:302] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b596f449af84e9cb30c84bf18f0242d, b3dce68ea6c1420d9819765b021c186b; no voters:
I20260928 10:56:29.202418 26410 raft_consensus.cc:2781] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:56:29.202780 26410 raft_consensus.cc:482] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:56:29.202944 26410 raft_consensus.cc:3037] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:29.205731 26410 raft_consensus.cc:504] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.206586 26410 leader_election.cc:288] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [CANDIDATE]: Term 1 election: Requested vote from peers d9d24f3b44984ceba67cf5c266287467 (127.5.227.130:36473), b3dce68ea6c1420d9819765b021c186b (127.5.227.131:45569)
I20260928 10:56:29.207396 26208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800630b22322449c866617388bb987fd" candidate_uuid: "8b596f449af84e9cb30c84bf18f0242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d24f3b44984ceba67cf5c266287467"
I20260928 10:56:29.207418 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800630b22322449c866617388bb987fd" candidate_uuid: "8b596f449af84e9cb30c84bf18f0242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dce68ea6c1420d9819765b021c186b"
I20260928 10:56:29.207695 26208 raft_consensus.cc:3037] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:29.207798 26339 raft_consensus.cc:3037] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:29.210186 26208 raft_consensus.cc:2454] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b596f449af84e9cb30c84bf18f0242d in term 1.
I20260928 10:56:29.210188 26339 raft_consensus.cc:2454] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b596f449af84e9cb30c84bf18f0242d in term 1.
I20260928 10:56:29.210807 26011 leader_election.cc:302] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b596f449af84e9cb30c84bf18f0242d, b3dce68ea6c1420d9819765b021c186b, d9d24f3b44984ceba67cf5c266287467; no voters:
I20260928 10:56:29.211231 26410 raft_consensus.cc:2781] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:29.212085 26410 raft_consensus.cc:686] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 1 LEADER]: Becoming Leader. State: Replica: 8b596f449af84e9cb30c84bf18f0242d, State: Running, Role: LEADER
I20260928 10:56:29.212719 26410 consensus_queue.cc:237] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } }
I20260928 10:56:29.218665 25936 catalog_manager.cc:5495] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b596f449af84e9cb30c84bf18f0242d (127.5.227.129). New cstate: current_term: 1 leader_uuid: "8b596f449af84e9cb30c84bf18f0242d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b596f449af84e9cb30c84bf18f0242d" member_type: VOTER last_known_addr { host: "127.5.227.129" port: 42099 } health_report { overall_health: HEALTHY } } }
W20260928 10:56:29.263371 26123 tablet.cc:2369] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:56:29.288436 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.281862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.5.227.129:42099)
0928 10:56:29.282182 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:29.282188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:29.282514 (+ 326us) server_negotiation.cc:407] Connection header received
0928 10:56:29.282629 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:29.282634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:29.282760 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:29.282989 (+ 229us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:56:29.283346 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.284263 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.285790 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.286283 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.286397 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.287083 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:56:29.287673 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:56:29.287894 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:29.288030 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 10:56:29.288113 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:56:29.298077 26339 raft_consensus.cc:1261] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 1 FOLLOWER]: Refusing update from remote peer 8b596f449af84e9cb30c84bf18f0242d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:29.298322 26208 raft_consensus.cc:1261] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 1 FOLLOWER]: Refusing update from remote peer 8b596f449af84e9cb30c84bf18f0242d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:56:29.299567 26414 consensus_queue.cc:1035] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d24f3b44984ceba67cf5c266287467" member_type: VOTER last_known_addr { host: "127.5.227.130" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:29.300163 26410 consensus_queue.cc:1035] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dce68ea6c1420d9819765b021c186b" member_type: VOTER last_known_addr { host: "127.5.227.131" port: 45569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:56:29.332616 25936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:29.332949 25936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:56:29.338831 25936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf00a05d12c1419da008efae4691a63b: Sending DeleteTablet for 3 replicas of tablet 800630b22322449c866617388bb987fd
I20260928 10:56:29.340548 26188 tablet_service.cc:1514] Processing DeleteTablet for tablet 800630b22322449c866617388bb987fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:29 UTC) from {username='slave'} at 127.0.0.1:39706
I20260928 10:56:29.341315 26319 tablet_service.cc:1514] Processing DeleteTablet for tablet 800630b22322449c866617388bb987fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:29 UTC) from {username='slave'} at 127.0.0.1:47666
I20260928 10:56:29.341614 26057 tablet_service.cc:1514] Processing DeleteTablet for tablet 800630b22322449c866617388bb987fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:56:29 UTC) from {username='slave'} at 127.0.0.1:43978
I20260928 10:56:29.342975 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25995
I20260928 10:56:29.353196 26426 tablet_replica.cc:330] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: stopping tablet replica
I20260928 10:56:29.353770 26426 raft_consensus.cc:2229] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:29.354385 26426 raft_consensus.cc:2258] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:29.363323 26426 ts_tablet_manager.cc:1905] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:29.364284 26428 tablet_replica.cc:330] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: stopping tablet replica
I20260928 10:56:29.365607 26428 raft_consensus.cc:2229] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:56:29.368971 26427 tablet_replica.cc:330] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: stopping tablet replica
I20260928 10:56:29.369164 26428 raft_consensus.cc:2258] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:29.370167 26427 raft_consensus.cc:2229] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:56:29.370745 26427 raft_consensus.cc:2258] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:56:29.372124 26426 ts_tablet_manager.cc:1918] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:29.372319 26426 log.cc:1196] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-1/wal/wals/800630b22322449c866617388bb987fd
I20260928 10:56:29.372915 26426 ts_tablet_manager.cc:1939] T 800630b22322449c866617388bb987fd P d9d24f3b44984ceba67cf5c266287467: Deleting consensus metadata
I20260928 10:56:29.373281 26427 ts_tablet_manager.cc:1905] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:29.373721 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.344697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.5.227.129:42099)
0928 10:56:29.344900 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:29.344907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:29.358823 (+ 13916us) server_negotiation.cc:407] Connection header received
0928 10:56:29.359447 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:29.359453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:29.359575 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:29.359665 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:29.360460 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.362066 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.366029 (+ 3963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.366919 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.367054 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.369777 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:29.369792 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:29.369809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:29.371170 (+ 1361us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:29.371249 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:29.371283 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:29.371428 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:29.373177 (+ 1749us) server_negotiation.cc:299] Negotiation successful
0928 10:56:29.373322 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:56:29.375018 25922 catalog_manager.cc:4841] TS d9d24f3b44984ceba67cf5c266287467 (127.5.227.130:36473): tablet 800630b22322449c866617388bb987fd (table test-table [id=292a209663e0450b95b2115f7c5e0ebf]) successfully deleted
I20260928 10:56:29.375388 26117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:29.383036 26427 ts_tablet_manager.cc:1918] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:56:29.383236 26427 log.cc:1196] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Deleting WAL directory at /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592619723274-6030-0/minicluster-data/ts-2/wal/wals/800630b22322449c866617388bb987fd
I20260928 10:56:29.383713 26427 ts_tablet_manager.cc:1939] T 800630b22322449c866617388bb987fd P b3dce68ea6c1420d9819765b021c186b: Deleting consensus metadata
I20260928 10:56:29.385739 25921 catalog_manager.cc:4841] TS b3dce68ea6c1420d9819765b021c186b (127.5.227.131:45569): tablet 800630b22322449c866617388bb987fd (table test-table [id=292a209663e0450b95b2115f7c5e0ebf]) successfully deleted
I20260928 10:56:29.376621 26428 ts_tablet_manager.cc:1905] T 800630b22322449c866617388bb987fd P 8b596f449af84e9cb30c84bf18f0242d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:56:29.483696 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25995
I20260928 10:56:29.503631 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26126
W20260928 10:56:29.503913 25922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.227.129:42099 (error 108)
I20260928 10:56:29.506129 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.505167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44002)
0928 10:56:29.505733 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.505936 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":311,"threads_started":1}
W20260928 10:56:29.506422 25922 catalog_manager.cc:4584] TS 8b596f449af84e9cb30c84bf18f0242d (127.5.227.129:42099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 800630b22322449c866617388bb987fd: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
I20260928 10:56:29.512305 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.504700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.5.227.130:36473)
0928 10:56:29.504923 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:29.504931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:29.505105 (+ 174us) server_negotiation.cc:407] Connection header received
0928 10:56:29.505333 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:29.505337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:29.505422 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:29.505700 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:29.506176 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.507232 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.508795 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.509224 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.509362 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.510752 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:29.510762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:29.510777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:29.511418 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:29.511473 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:29.511495 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:29.511636 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:29.511809 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 10:56:29.511942 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:56:29.513216 26248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:29.560801 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.560251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44006)
0928 10:56:29.560498 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.560591 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 10:56:29.561931 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.561513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44010)
0928 10:56:29.561677 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.561786 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 10:56:29.621364 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 26126
I20260928 10:56:29.624991 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.624365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44016)
0928 10:56:29.624642 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.624758 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 10:56:29.626382 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.625853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44028)
0928 10:56:29.626038 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.626152 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 10:56:29.637784 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26257
I20260928 10:56:29.646497 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.638707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.5.227.131:45569)
0928 10:56:29.638932 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:29.638941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:29.638965 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:56:29.639281 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:29.639287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:29.639426 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:29.639695 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:29.640231 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.640962 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.642300 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:29.642939 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:29.643165 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:29.644869 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:29.644883 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:29.644905 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:29.645547 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:29.645606 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:29.645635 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:29.645797 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:29.645990 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 10:56:29.646131 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 10:56:29.647240 26379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:29.708027 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.707366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44044)
0928 10:56:29.707655 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.707820 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 10:56:29.709491 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.708891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44048)
0928 10:56:29.709199 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.709296 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260928 10:56:29.747500 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 26257
I20260928 10:56:29.766790 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 25905
I20260928 10:56:29.768271 25966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:29.890383 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:29.889467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.129:42099 (local address 127.0.0.1:44062)
0928 10:56:29.889898 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 10:56:29.890079 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.129:42099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 10:56:29.890627 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 25905
2026-09-28T10:56:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3395 ms)
[----------] 6 tests from EncryptionPolicyTest (25131 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:30 dist-test-slave-8w8g krb5kdc[26438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:30 dist-test-slave-8w8g krb5kdc[26438](info): set up 2 sockets
Sep 28 10:56:30 dist-test-slave-8w8g krb5kdc[26438](info): commencing operation
krb5kdc: starting...
W20260928 10:56:31.104784 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:31 dist-test-slave-8w8g krb5kdc[26438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:31.417039 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:31.811293 26451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:31.811966 26451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:31.812245 26451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:31.822031 26451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:31.822185 26451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:31.822270 26451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:31.822365 26451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:31.822428 26451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:31.834376 26451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:31.836412 26451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:31.838035 26451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:31.855772 26456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:31.855647 26459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:31.855783 26457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:31.856875 26451 server_base.cc:1034] running on GCE node
I20260928 10:56:31.857865 26451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:31.859854 26451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:31.861131 26451 hybrid_clock.cc:648] HybridClock initialized: now 1790592991861067 us; error 60 us; skew 500 ppm
I20260928 10:56:31.861613 26451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:31 dist-test-slave-8w8g krb5kdc[26438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:31.866180 26451 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:31.868396 26451 webserver.cc:460] Webserver started at http://127.0.0.1:32851/ using document root <none> and password file <none>
I20260928 10:56:31.869242 26451 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:31.869365 26451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:31.869760 26451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:31.873283 26451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "dee68a012aec407bbbcd41c057de9c16"
format_stamp: "Formatted at 2026-09-28 10:56:31 on dist-test-slave-8w8g"
server_key: "a1ce70c8c0944634d6322eead632032b"
server_key_iv: "81e27cf4694bbc493ddc277d38b9a3e2"
server_key_version: "encryptionkey@0"
I20260928 10:56:31.874243 26451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "dee68a012aec407bbbcd41c057de9c16"
format_stamp: "Formatted at 2026-09-28 10:56:31 on dist-test-slave-8w8g"
server_key: "a1ce70c8c0944634d6322eead632032b"
server_key_iv: "81e27cf4694bbc493ddc277d38b9a3e2"
server_key_version: "encryptionkey@0"
I20260928 10:56:31.881397 26451 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260928 10:56:31.886413 26466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:31.888528 26451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:56:31.889066 26451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "dee68a012aec407bbbcd41c057de9c16"
format_stamp: "Formatted at 2026-09-28 10:56:31 on dist-test-slave-8w8g"
server_key: "a1ce70c8c0944634d6322eead632032b"
server_key_iv: "81e27cf4694bbc493ddc277d38b9a3e2"
server_key_version: "encryptionkey@0"
I20260928 10:56:31.889386 26451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:31.922385 26451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:31.929348 26451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:31.929719 26451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:31.957928 26451 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42353
I20260928 10:56:31.957933 26517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42353 every 8 connection(s)
I20260928 10:56:31.959616 26451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:31.964138 26518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:31.969669 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 26451
I20260928 10:56:31.969933 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:31.971081 6030 external_mini_cluster.cc:1428] Setting key 8be45ae2eabe6c1efc1804c0fc182901
I20260928 10:56:31.982633 26518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16: Bootstrap starting.
I20260928 10:56:31.986716 26518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:31.988765 26518 log.cc:824] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:56:31 dist-test-slave-8w8g krb5kdc[26438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:56:31.994866 26518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16: No bootstrap required, opened a new log
I20260928 10:56:32.003188 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:31.974970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34159 (local address 127.0.0.1:42353)
0928 10:56:31.976344 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:31.976358 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:31.976410 (+ 52us) server_negotiation.cc:407] Connection header received
0928 10:56:31.978523 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:31.978568 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:31.979109 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:31.979670 (+ 561us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:31.981415 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:31.983103 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:31.985186 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:31.986063 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:31.986334 (+ 271us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:31.992060 (+ 5726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:31.992071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:31.992093 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:31.992141 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:31.997071 (+ 4930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:31.998523 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:31.998532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:31.998541 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:31.998870 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:31.999506 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:31.999511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:31.999513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:32.000255 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:32.000695 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:32.001352 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 10:56:32.001905 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":698,"threads_started":1}
I20260928 10:56:32.005190 26518 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee68a012aec407bbbcd41c057de9c16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260928 10:56:32.005648 26518 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:32.005780 26518 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee68a012aec407bbbcd41c057de9c16, State: Initialized, Role: FOLLOWER
I20260928 10:56:32.006850 26518 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee68a012aec407bbbcd41c057de9c16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260928 10:56:32.007198 26518 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:32.007371 26518 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:32.007578 26518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:32.009326 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:32.010411 26518 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee68a012aec407bbbcd41c057de9c16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260928 10:56:32.011019 26518 leader_election.cc:302] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee68a012aec407bbbcd41c057de9c16; no voters:
I20260928 10:56:32.011781 26518 leader_election.cc:288] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:32.011878 26523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:32.014789 26518 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:32.016121 26523 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [term 1 LEADER]: Becoming Leader. State: Replica: dee68a012aec407bbbcd41c057de9c16, State: Running, Role: LEADER
I20260928 10:56:32.017055 26523 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee68a012aec407bbbcd41c057de9c16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260928 10:56:32.021935 26525 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee68a012aec407bbbcd41c057de9c16. Latest consensus state: current_term: 1 leader_uuid: "dee68a012aec407bbbcd41c057de9c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee68a012aec407bbbcd41c057de9c16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260928 10:56:32.022521 26525 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:32.023253 26524 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee68a012aec407bbbcd41c057de9c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee68a012aec407bbbcd41c057de9c16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260928 10:56:32.023846 26524 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:32.024354 26534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:32.040958 26534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:32.065896 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:32.031174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46345 (local address 127.0.0.1:42353)
0928 10:56:32.031425 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:32.031435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:32.034051 (+ 2616us) server_negotiation.cc:407] Connection header received
0928 10:56:32.034320 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:32.034329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:32.034465 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:32.034640 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:32.037447 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:32.039910 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:32.042038 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:32.042703 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:32.042838 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:32.049094 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:32.049103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:32.049108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:32.049176 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:32.056545 (+ 7369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:32.058476 (+ 1931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:32.058487 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:32.058494 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:32.058756 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:32.059771 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:32.059780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:32.059786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:32.060708 (+ 922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:32.063720 (+ 3012us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:32.064940 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0928 10:56:32.065296 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:56:32.072948 26534 catalog_manager.cc:1347] Generated new cluster ID: 3314159982f048b5a5fd7642986625df
I20260928 10:56:32.073088 26534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:32.096555 26534 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:32.098294 26534 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:32.108889 26534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee68a012aec407bbbcd41c057de9c16: Generated new TSK 0
I20260928 10:56:32.110219 26534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:32.118875 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26451
I20260928 10:56:32.120349 26513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:32.221675 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 26451
W20260928 10:56:32.240067 26436 connection.cc:535] client connection to 127.0.0.1:42353 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42353): Connection reset by peer (error 104)
2026-09-28T10:56:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:32 dist-test-slave-8w8g krb5kdc[26550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:32 dist-test-slave-8w8g krb5kdc[26550](info): set up 2 sockets
Sep 28 10:56:32 dist-test-slave-8w8g krb5kdc[26550](info): commencing operation
krb5kdc: starting...
W20260928 10:56:33.493417 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:33 dist-test-slave-8w8g krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:33.820691 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:34.207916 26563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:34.208379 26563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:34.208623 26563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:34.216847 26563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:34.216992 26563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:34.217039 26563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:34.217093 26563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:34.217131 26563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:34.229004 26563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:34.230574 26563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:34.232021 26563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:34.246007 26568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:34.245923 26569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:34.246042 26571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:34.248158 26563 server_base.cc:1034] running on GCE node
I20260928 10:56:34.249181 26563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:34.250923 26563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:34.252221 26563 hybrid_clock.cc:648] HybridClock initialized: now 1790592994252140 us; error 63 us; skew 500 ppm
I20260928 10:56:34.252779 26563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:34 dist-test-slave-8w8g krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:34.262372 26563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:34.264871 26563 webserver.cc:460] Webserver started at http://127.0.0.1:33423/ using document root <none> and password file <none>
I20260928 10:56:34.265758 26563 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:34.265942 26563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:34.266305 26563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:34.270125 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "133a6875f0994b33aadd5de648ab3d20"
format_stamp: "Formatted at 2026-09-28 10:56:34 on dist-test-slave-8w8g"
server_key: "57f573e7d49243530a5f20b3fd1ee0ca"
server_key_iv: "04706f992eaf978118b719f0d4f2417c"
server_key_version: "encryptionkey@0"
I20260928 10:56:34.271096 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "133a6875f0994b33aadd5de648ab3d20"
format_stamp: "Formatted at 2026-09-28 10:56:34 on dist-test-slave-8w8g"
server_key: "57f573e7d49243530a5f20b3fd1ee0ca"
server_key_iv: "04706f992eaf978118b719f0d4f2417c"
server_key_version: "encryptionkey@0"
I20260928 10:56:34.278298 26563 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260928 10:56:34.283133 26578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:34.285037 26563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 10:56:34.285238 26563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "133a6875f0994b33aadd5de648ab3d20"
format_stamp: "Formatted at 2026-09-28 10:56:34 on dist-test-slave-8w8g"
server_key: "57f573e7d49243530a5f20b3fd1ee0ca"
server_key_iv: "04706f992eaf978118b719f0d4f2417c"
server_key_version: "encryptionkey@0"
I20260928 10:56:34.285499 26563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:34.314891 26563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:34.320679 26563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:34.320997 26563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:34.348795 26563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44727
I20260928 10:56:34.348836 26629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44727 every 8 connection(s)
I20260928 10:56:34.350793 26563 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:34.353163 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 26563
I20260928 10:56:34.353315 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:34.354053 6030 external_mini_cluster.cc:1428] Setting key 7ddf59cdfeb8697920750a99d734cae0
I20260928 10:56:34.355587 26630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:34.372815 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:34.356469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40925 (local address 127.0.0.1:44727)
0928 10:56:34.357334 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:34.357351 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:34.357410 (+ 59us) server_negotiation.cc:407] Connection header received
0928 10:56:34.359489 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:34.359568 (+ 79us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:34.360249 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:34.360881 (+ 632us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:34.361734 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:34.364401 (+ 2667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:34.365964 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:34.367271 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:34.367509 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:34.369367 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:34.369386 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:34.369417 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:34.370296 (+ 879us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:34.370416 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:34.370464 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:34.370658 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:34.371113 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 10:56:34.371323 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":363,"threads_started":1}
I20260928 10:56:34.372893 26630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20: Bootstrap starting.
I20260928 10:56:34.376807 26630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:34.378186 26630 log.cc:824] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:34.378928 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:34.381844 26630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20: No bootstrap required, opened a new log
I20260928 10:56:34.393738 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:34.384243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40777 (local address 127.0.0.1:44727)
0928 10:56:34.384481 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:34.384488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:34.385172 (+ 684us) server_negotiation.cc:407] Connection header received
0928 10:56:34.385421 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:34.385426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:34.385520 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:34.385621 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:34.386387 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:34.387724 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:34.389924 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:34.390571 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:34.390794 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:34.392421 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:34.392426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:34.392431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:34.392571 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:34.392631 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:34.392654 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:34.392777 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:34.393045 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 10:56:34.393146 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:56:34.394107 26630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133a6875f0994b33aadd5de648ab3d20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44727 } }
I20260928 10:56:34.394673 26630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:34.394793 26630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133a6875f0994b33aadd5de648ab3d20, State: Initialized, Role: FOLLOWER
I20260928 10:56:34.395565 26630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133a6875f0994b33aadd5de648ab3d20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44727 } }
I20260928 10:56:34.395869 26630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:34.396034 26630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:34.396222 26630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:34.398768 26630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133a6875f0994b33aadd5de648ab3d20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44727 } }
I20260928 10:56:34.399365 26630 leader_election.cc:302] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133a6875f0994b33aadd5de648ab3d20; no voters:
I20260928 10:56:34.400372 26630 leader_election.cc:288] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:34.400405 26640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:34.402014 26640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [term 1 LEADER]: Becoming Leader. State: Replica: 133a6875f0994b33aadd5de648ab3d20, State: Running, Role: LEADER
I20260928 10:56:34.402632 26640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133a6875f0994b33aadd5de648ab3d20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44727 } }
I20260928 10:56:34.403636 26630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:34.406134 26642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "133a6875f0994b33aadd5de648ab3d20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133a6875f0994b33aadd5de648ab3d20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44727 } } }
I20260928 10:56:34.406798 26642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:34.409795 26645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:34.409763 26641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133a6875f0994b33aadd5de648ab3d20. Latest consensus state: current_term: 1 leader_uuid: "133a6875f0994b33aadd5de648ab3d20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133a6875f0994b33aadd5de648ab3d20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44727 } } }
I20260928 10:56:34.410379 26641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:34.419278 26645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:34.437106 26645 catalog_manager.cc:1347] Generated new cluster ID: 0ad444eb8dc9492d8169a257157f7c06
I20260928 10:56:34.437245 26645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:34.461108 26645 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:34.462880 26645 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:34.475098 26645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 133a6875f0994b33aadd5de648ab3d20: Generated new TSK 0
I20260928 10:56:34.476477 26645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:34.485687 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26563
W20260928 10:56:34.487303 26625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40925
W20260928 10:56:34.488462 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:34.488706 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 26563
W20260928 10:56:34.507282 26548 connection.cc:535] client connection to 127.0.0.1:44727 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44727): Connection reset by peer (error 104)
2026-09-28T10:56:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:34 dist-test-slave-8w8g krb5kdc[26662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:34 dist-test-slave-8w8g krb5kdc[26662](info): set up 2 sockets
Sep 28 10:56:34 dist-test-slave-8w8g krb5kdc[26662](info): commencing operation
krb5kdc: starting...
W20260928 10:56:35.709292 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:35 dist-test-slave-8w8g krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:36.013489 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:36.394727 26675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:36.395160 26675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:36.395330 26675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:36.403669 26675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:36.403887 26675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:36.404004 26675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:36.404116 26675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:36.404194 26675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:36.415031 26675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:36.416858 26675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:36.418541 26675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:36.434659 26680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:36.434589 26681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:36.434644 26683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:36.434922 26675 server_base.cc:1034] running on GCE node
I20260928 10:56:36.436201 26675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:36.437872 26675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:36.439217 26675 hybrid_clock.cc:648] HybridClock initialized: now 1790592996439139 us; error 76 us; skew 500 ppm
I20260928 10:56:36.439932 26675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:36 dist-test-slave-8w8g krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:36.443827 26675 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:36.445536 26675 webserver.cc:460] Webserver started at http://127.0.0.1:33273/ using document root <none> and password file <none>
I20260928 10:56:36.446256 26675 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:36.446378 26675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:36.446820 26675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:36.450341 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "4c86d9caa9d84d7ab823c87d85777a84"
format_stamp: "Formatted at 2026-09-28 10:56:36 on dist-test-slave-8w8g"
server_key: "5aa03d78742b3b56f5b644f6a7d55215"
server_key_iv: "4833936cb8bf75f35953c4e916d014af"
server_key_version: "encryptionkey@0"
I20260928 10:56:36.451184 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "4c86d9caa9d84d7ab823c87d85777a84"
format_stamp: "Formatted at 2026-09-28 10:56:36 on dist-test-slave-8w8g"
server_key: "5aa03d78742b3b56f5b644f6a7d55215"
server_key_iv: "4833936cb8bf75f35953c4e916d014af"
server_key_version: "encryptionkey@0"
I20260928 10:56:36.457549 26675 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260928 10:56:36.461908 26690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:36.463775 26675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:56:36.464005 26675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "4c86d9caa9d84d7ab823c87d85777a84"
format_stamp: "Formatted at 2026-09-28 10:56:36 on dist-test-slave-8w8g"
server_key: "5aa03d78742b3b56f5b644f6a7d55215"
server_key_iv: "4833936cb8bf75f35953c4e916d014af"
server_key_version: "encryptionkey@0"
I20260928 10:56:36.464237 26675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:36.471469 26675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:36.477161 26675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:36.477684 26675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:36.501152 26675 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45357
I20260928 10:56:36.501187 26741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45357 every 8 connection(s)
I20260928 10:56:36.503005 26675 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:36.503786 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 26675
I20260928 10:56:36.504019 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:36.504665 6030 external_mini_cluster.cc:1428] Setting key 708a17525e01117cdf9c6edc8dff783f
I20260928 10:56:36.508213 26742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:36.513689 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:36.506664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37321 (local address 127.0.0.1:45357)
0928 10:56:36.507556 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:36.507569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:36.507617 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:56:36.509754 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:36.509778 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:36.510165 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:36.510600 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:36.511799 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:36.511808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:36.511832 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:36.511848 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:36.511934 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:36.511961 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:36.512213 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:36.512364 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 10:56:36.512567 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":347,"threads_started":1}
I20260928 10:56:36.518560 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:36.521567 26742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84: Bootstrap starting.
I20260928 10:56:36.525693 26742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:36.526937 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:36.523509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36823 (local address 127.0.0.1:45357)
0928 10:56:36.523788 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:36.523798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:36.524329 (+ 531us) server_negotiation.cc:407] Connection header received
0928 10:56:36.524596 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:36.524601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:36.524661 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:36.524773 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:36.526050 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:36.526055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:36.526058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:36.526076 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:36.526129 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:36.526146 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:36.526260 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:36.526487 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 10:56:36.526597 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:56:36.527217 26742 log.cc:824] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:36.530742 26742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84: No bootstrap required, opened a new log
I20260928 10:56:36.539369 26742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86d9caa9d84d7ab823c87d85777a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45357 } }
I20260928 10:56:36.539911 26742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:36.540035 26742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c86d9caa9d84d7ab823c87d85777a84, State: Initialized, Role: FOLLOWER
I20260928 10:56:36.540792 26742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86d9caa9d84d7ab823c87d85777a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45357 } }
I20260928 10:56:36.541121 26742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:36.541294 26742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:36.541478 26742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:36.544379 26742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86d9caa9d84d7ab823c87d85777a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45357 } }
I20260928 10:56:36.545209 26742 leader_election.cc:302] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c86d9caa9d84d7ab823c87d85777a84; no voters:
I20260928 10:56:36.545904 26742 leader_election.cc:288] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:36.546105 26752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:36.547233 26752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [term 1 LEADER]: Becoming Leader. State: Replica: 4c86d9caa9d84d7ab823c87d85777a84, State: Running, Role: LEADER
I20260928 10:56:36.547900 26752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86d9caa9d84d7ab823c87d85777a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45357 } }
I20260928 10:56:36.548734 26742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:36.551235 26754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c86d9caa9d84d7ab823c87d85777a84. Latest consensus state: current_term: 1 leader_uuid: "4c86d9caa9d84d7ab823c87d85777a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86d9caa9d84d7ab823c87d85777a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45357 } } }
I20260928 10:56:36.551229 26753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c86d9caa9d84d7ab823c87d85777a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c86d9caa9d84d7ab823c87d85777a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45357 } } }
I20260928 10:56:36.551841 26753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:36.551841 26754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:36.557569 26757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:36.562958 26757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:36.572929 26757 catalog_manager.cc:1347] Generated new cluster ID: f33bfc5fc92e45658acbaf3a3b530608
I20260928 10:56:36.573078 26757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:36.603091 26757 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:36.603714 26757 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:36.614719 26757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c86d9caa9d84d7ab823c87d85777a84: Generated new TSK 0
I20260928 10:56:36.615814 26757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:36.625496 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26675
W20260928 10:56:36.626757 26737 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37321
W20260928 10:56:36.627954 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26675: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:36.628156 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 26675
2026-09-28T10:56:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2136 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:36 dist-test-slave-8w8g krb5kdc[26773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:36 dist-test-slave-8w8g krb5kdc[26773](info): set up 2 sockets
Sep 28 10:56:36 dist-test-slave-8w8g krb5kdc[26773](info): commencing operation
krb5kdc: starting...
W20260928 10:56:37.825021 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:38 dist-test-slave-8w8g krb5kdc[26773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:38.195323 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:38.662650 26786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:38.663080 26786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:38.663188 26786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:38.671494 26786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:38.671659 26786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:38.671708 26786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:38.671883 26786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:38.671943 26786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:38.684471 26786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:38.686075 26786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:38.687512 26786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:38.704267 26791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:38.704309 26792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:38.704829 26794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:38.705798 26786 server_base.cc:1034] running on GCE node
I20260928 10:56:38.706913 26786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:38.708973 26786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:38.710368 26786 hybrid_clock.cc:648] HybridClock initialized: now 1790592998710336 us; error 177 us; skew 500 ppm
I20260928 10:56:38.710996 26786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:38 dist-test-slave-8w8g krb5kdc[26773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:38.715001 26786 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:38.717229 26786 webserver.cc:460] Webserver started at http://127.0.0.1:40957/ using document root <none> and password file <none>
I20260928 10:56:38.718403 26786 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:38.718997 26786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:38.719432 26786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:38.723225 26786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de"
format_stamp: "Formatted at 2026-09-28 10:56:38 on dist-test-slave-8w8g"
server_key: "5ff88ec2845b99e27fe953357a3770cf"
server_key_iv: "66d1c2c59344f6ce12e7eaa8500bfdd2"
server_key_version: "encryptionkey@0"
I20260928 10:56:38.725754 26786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de"
format_stamp: "Formatted at 2026-09-28 10:56:38 on dist-test-slave-8w8g"
server_key: "5ff88ec2845b99e27fe953357a3770cf"
server_key_iv: "66d1c2c59344f6ce12e7eaa8500bfdd2"
server_key_version: "encryptionkey@0"
I20260928 10:56:38.733515 26786 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260928 10:56:38.738348 26801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:38.740576 26786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:56:38.740878 26786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de"
format_stamp: "Formatted at 2026-09-28 10:56:38 on dist-test-slave-8w8g"
server_key: "5ff88ec2845b99e27fe953357a3770cf"
server_key_iv: "66d1c2c59344f6ce12e7eaa8500bfdd2"
server_key_version: "encryptionkey@0"
I20260928 10:56:38.741160 26786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:38.774530 26786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:38.781281 26786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:38.781775 26786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:38.807653 26786 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43645
I20260928 10:56:38.807813 26852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43645 every 8 connection(s)
I20260928 10:56:38.810527 26786 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:38.813443 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 26786
I20260928 10:56:38.813889 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:38.814862 6030 external_mini_cluster.cc:1428] Setting key 75d2a4e8ae71b3c855c3791f501d5ae5
I20260928 10:56:38.815542 26853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:38.835865 26853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de: Bootstrap starting.
Sep 28 10:56:38 dist-test-slave-8w8g krb5kdc[26773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:56:38.842216 26853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:38.843914 26853 log.cc:824] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:38.849098 26853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de: No bootstrap required, opened a new log
I20260928 10:56:38.851468 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:38.817256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49721 (local address 127.0.0.1:43645)
0928 10:56:38.818474 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:38.818487 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:38.818541 (+ 54us) server_negotiation.cc:407] Connection header received
0928 10:56:38.822237 (+ 3696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:38.822271 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:38.822876 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:38.823326 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:38.826024 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:38.827520 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:38.829643 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:38.831019 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:38.831215 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:38.838739 (+ 7524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:38.838754 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:38.838779 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:38.838845 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:38.844136 (+ 5291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:38.845621 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:38.845630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:38.845637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:38.845936 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:38.846501 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:38.846506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:38.846509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:38.847313 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:38.847975 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:38.848325 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 10:56:38.849218 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":713,"threads_started":1}
I20260928 10:56:38.857098 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:38.860219 26853 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43645 } }
I20260928 10:56:38.860857 26853 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:38.861001 26853 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec990c9f2cc8419a975fa5b4fc7bb5de, State: Initialized, Role: FOLLOWER
I20260928 10:56:38.861707 26853 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43645 } }
I20260928 10:56:38.862052 26853 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:38.862221 26853 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:38.862705 26853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:38.865624 26853 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43645 } }
I20260928 10:56:38.866304 26853 leader_election.cc:302] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec990c9f2cc8419a975fa5b4fc7bb5de; no voters:
I20260928 10:56:38.867056 26853 leader_election.cc:288] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:38.867208 26863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:38.870371 26853 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:38.873783 26863 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [term 1 LEADER]: Becoming Leader. State: Replica: ec990c9f2cc8419a975fa5b4fc7bb5de, State: Running, Role: LEADER
I20260928 10:56:38.875326 26863 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43645 } }
I20260928 10:56:38.881278 26865 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec990c9f2cc8419a975fa5b4fc7bb5de. Latest consensus state: current_term: 1 leader_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43645 } } }
I20260928 10:56:38.881256 26864 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec990c9f2cc8419a975fa5b4fc7bb5de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43645 } } }
I20260928 10:56:38.882094 26865 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:38.882094 26864 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:38.886915 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:38.862796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38259 (local address 127.0.0.1:43645)
0928 10:56:38.863084 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:38.863094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:38.863396 (+ 302us) server_negotiation.cc:407] Connection header received
0928 10:56:38.863652 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:38.863662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:38.863795 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:38.863922 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:38.865707 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:38.866979 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:38.869018 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:38.869580 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:38.869746 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:38.872644 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:38.872651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:38.872659 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:38.872713 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:38.879177 (+ 6464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:38.880891 (+ 1714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:38.881056 (+ 165us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:38.881065 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:38.881351 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:38.882277 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:38.882284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:38.882288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:38.883602 (+ 1314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:38.885671 (+ 2069us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:38.885953 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 10:56:38.886315 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 10:56:38.891229 26872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:38.898411 26872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:38.912984 26872 catalog_manager.cc:1347] Generated new cluster ID: 542b0a101dbc4cd18b32f2414cff8e95
I20260928 10:56:38.913153 26872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:38.932281 26872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:38.934001 26872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:38.945190 26872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec990c9f2cc8419a975fa5b4fc7bb5de: Generated new TSK 0
I20260928 10:56:38.947052 26872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:38.962857 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26786
I20260928 10:56:38.964625 26848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:39.072451 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 26786
2026-09-28T10:56:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:39 dist-test-slave-8w8g krb5kdc[26885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:39 dist-test-slave-8w8g krb5kdc[26885](info): set up 2 sockets
Sep 28 10:56:39 dist-test-slave-8w8g krb5kdc[26885](info): commencing operation
krb5kdc: starting...
W20260928 10:56:40.381243 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:40 dist-test-slave-8w8g krb5kdc[26885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:40.717133 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:41.078758 26898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:41.079166 26898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:41.079344 26898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:41.087455 26898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:41.087683 26898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:41.087822 26898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:41.087949 26898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:41.088037 26898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:41.100126 26898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:41.101733 26898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:41.103339 26898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:41.119809 26903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:41.120044 26906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:41.120122 26904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:41.121483 26898 server_base.cc:1034] running on GCE node
I20260928 10:56:41.122388 26898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:41.124097 26898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:41.125344 26898 hybrid_clock.cc:648] HybridClock initialized: now 1790593001125265 us; error 79 us; skew 500 ppm
I20260928 10:56:41.125756 26898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:41 dist-test-slave-8w8g krb5kdc[26885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:41.136261 26898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:41.138181 26898 webserver.cc:460] Webserver started at http://127.0.0.1:41419/ using document root <none> and password file <none>
I20260928 10:56:41.139024 26898 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:41.139207 26898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:41.139580 26898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:41.143564 26898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "75082586bb9c461985f773ccd2469cd2"
format_stamp: "Formatted at 2026-09-28 10:56:41 on dist-test-slave-8w8g"
server_key: "6d67b327085ac817dca16ffe85ab872b"
server_key_iv: "222b1cd4d0d8b26dcb8bf8e7a70da595"
server_key_version: "encryptionkey@0"
I20260928 10:56:41.144601 26898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "75082586bb9c461985f773ccd2469cd2"
format_stamp: "Formatted at 2026-09-28 10:56:41 on dist-test-slave-8w8g"
server_key: "6d67b327085ac817dca16ffe85ab872b"
server_key_iv: "222b1cd4d0d8b26dcb8bf8e7a70da595"
server_key_version: "encryptionkey@0"
I20260928 10:56:41.151697 26898 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:56:41.157094 26913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:41.158977 26898 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 10:56:41.159236 26898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "75082586bb9c461985f773ccd2469cd2"
format_stamp: "Formatted at 2026-09-28 10:56:41 on dist-test-slave-8w8g"
server_key: "6d67b327085ac817dca16ffe85ab872b"
server_key_iv: "222b1cd4d0d8b26dcb8bf8e7a70da595"
server_key_version: "encryptionkey@0"
I20260928 10:56:41.159662 26898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:41.180819 26898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:41.186370 26898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:41.186972 26898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:41.213554 26898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45565
I20260928 10:56:41.213587 26964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45565 every 8 connection(s)
I20260928 10:56:41.215095 26898 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:41.217734 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 26898
I20260928 10:56:41.218165 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:41.218771 6030 external_mini_cluster.cc:1428] Setting key 474d990d2270e23df68b45d4af81ad01
I20260928 10:56:41.220165 26965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:41.236410 26965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2: Bootstrap starting.
I20260928 10:56:41.236627 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:41.220861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56979 (local address 127.0.0.1:45565)
0928 10:56:41.221879 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:41.221894 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:41.221952 (+ 58us) server_negotiation.cc:407] Connection header received
0928 10:56:41.224012 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:41.224043 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:41.224601 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:41.225081 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:41.225857 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:41.227649 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:41.229819 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:41.230669 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:41.230841 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:41.232426 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:41.232496 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:41.232576 (+ 80us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:41.233358 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:41.233565 (+ 207us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:41.233616 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:41.233735 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:41.234166 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 10:56:41.234744 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":459,"threads_started":1}
I20260928 10:56:41.240945 26965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:41.241786 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:41.242324 26965 log.cc:824] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:41.246526 26965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2: No bootstrap required, opened a new log
I20260928 10:56:41.256114 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:41.247416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47519 (local address 127.0.0.1:45565)
0928 10:56:41.247825 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:41.247832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:41.248478 (+ 646us) server_negotiation.cc:407] Connection header received
0928 10:56:41.248668 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:41.248674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:41.248763 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:41.248900 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:41.249471 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:41.250812 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:41.252616 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:41.253461 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:41.253618 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:41.255157 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:41.255167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:41.255173 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:41.255206 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:41.255265 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:41.255287 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:41.255404 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:41.255571 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 10:56:41.255683 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260928 10:56:41.258046 26965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082586bb9c461985f773ccd2469cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260928 10:56:41.258525 26965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:41.258657 26965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75082586bb9c461985f773ccd2469cd2, State: Initialized, Role: FOLLOWER
I20260928 10:56:41.259364 26965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082586bb9c461985f773ccd2469cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260928 10:56:41.259652 26965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:41.259868 26965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:41.260087 26965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:41.262877 26965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082586bb9c461985f773ccd2469cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260928 10:56:41.263566 26965 leader_election.cc:302] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75082586bb9c461985f773ccd2469cd2; no voters:
I20260928 10:56:41.264353 26965 leader_election.cc:288] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:41.264545 26975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:41.267083 26965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:41.268031 26975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 75082586bb9c461985f773ccd2469cd2, State: Running, Role: LEADER
I20260928 10:56:41.268805 26975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082586bb9c461985f773ccd2469cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260928 10:56:41.272383 26977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75082586bb9c461985f773ccd2469cd2. Latest consensus state: current_term: 1 leader_uuid: "75082586bb9c461985f773ccd2469cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082586bb9c461985f773ccd2469cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } } }
I20260928 10:56:41.272989 26977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:41.272847 26976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75082586bb9c461985f773ccd2469cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082586bb9c461985f773ccd2469cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } } }
I20260928 10:56:41.273208 26976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:41.275951 26985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:41.281123 26985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:41.291117 26985 catalog_manager.cc:1347] Generated new cluster ID: a43983d2f5f54e8495241fb0d2086720
I20260928 10:56:41.291252 26985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:41.311132 26985 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:41.312994 26985 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:41.325958 26985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75082586bb9c461985f773ccd2469cd2: Generated new TSK 0
I20260928 10:56:41.326933 26985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:41.345405 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26898
W20260928 10:56:41.346920 26960 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56979
W20260928 10:56:41.348074 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 26898: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:41.348280 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 26898
2026-09-28T10:56:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:41 dist-test-slave-8w8g krb5kdc[26996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:41 dist-test-slave-8w8g krb5kdc[26996](info): set up 2 sockets
Sep 28 10:56:41 dist-test-slave-8w8g krb5kdc[26996](info): commencing operation
krb5kdc: starting...
W20260928 10:56:42.565176 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:42 dist-test-slave-8w8g krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:42.874298 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:43.274708 27009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:43.275067 27009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:43.275390 27009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:43.284549 27009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:43.284693 27009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:43.284776 27009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:43.284875 27009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:43.284968 27009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:43.296288 27009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:43.297739 27009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:43.299175 27009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:43.313607 27014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:43.313632 27017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:43.313720 27015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:43.315214 27009 server_base.cc:1034] running on GCE node
I20260928 10:56:43.316457 27009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:43.318080 27009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:43.319348 27009 hybrid_clock.cc:648] HybridClock initialized: now 1790593003319265 us; error 73 us; skew 500 ppm
I20260928 10:56:43.319954 27009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:43 dist-test-slave-8w8g krb5kdc[26996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:43.323781 27009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:43.325882 27009 webserver.cc:460] Webserver started at http://127.0.0.1:44291/ using document root <none> and password file <none>
I20260928 10:56:43.326687 27009 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:43.326848 27009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:43.327196 27009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:43.330910 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "ead214bbd5ff48d987512c29ced1b5e4"
format_stamp: "Formatted at 2026-09-28 10:56:43 on dist-test-slave-8w8g"
server_key: "463674eedb84692b953a28199c76c9a2"
server_key_iv: "38e96d4e4e5ab901b5d4248df33f5522"
server_key_version: "encryptionkey@0"
I20260928 10:56:43.331970 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "ead214bbd5ff48d987512c29ced1b5e4"
format_stamp: "Formatted at 2026-09-28 10:56:43 on dist-test-slave-8w8g"
server_key: "463674eedb84692b953a28199c76c9a2"
server_key_iv: "38e96d4e4e5ab901b5d4248df33f5522"
server_key_version: "encryptionkey@0"
I20260928 10:56:43.338305 27009 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:56:43.343019 27024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:43.344978 27009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 10:56:43.345230 27009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "ead214bbd5ff48d987512c29ced1b5e4"
format_stamp: "Formatted at 2026-09-28 10:56:43 on dist-test-slave-8w8g"
server_key: "463674eedb84692b953a28199c76c9a2"
server_key_iv: "38e96d4e4e5ab901b5d4248df33f5522"
server_key_version: "encryptionkey@0"
I20260928 10:56:43.345639 27009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:43.353195 27009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:43.359278 27009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:43.359654 27009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:43.383986 27009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36969
I20260928 10:56:43.384030 27075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36969 every 8 connection(s)
I20260928 10:56:43.385634 27009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:43.386873 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27009
I20260928 10:56:43.387092 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:43.387712 6030 external_mini_cluster.cc:1428] Setting key 6c1c5ec4f1ae4301bf100233b65ce388
I20260928 10:56:43.391305 27076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:43.397085 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:43.389830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58061 (local address 127.0.0.1:36969)
0928 10:56:43.390861 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:43.390874 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:43.390919 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:56:43.393005 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:43.393031 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:43.393397 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:43.393826 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:43.395203 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:43.395210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:43.395223 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:43.395241 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:43.395327 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:43.395351 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:43.395628 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:43.395818 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 10:56:43.396028 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":430,"threads_started":1}
I20260928 10:56:43.402362 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:43.406014 27076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4: Bootstrap starting.
I20260928 10:56:43.411391 27076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:43.412348 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:43.407830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46285 (local address 127.0.0.1:36969)
0928 10:56:43.408084 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:43.408092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:43.408651 (+ 559us) server_negotiation.cc:407] Connection header received
0928 10:56:43.408901 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:43.408906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:43.408970 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:43.409073 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:43.410375 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:43.410380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:43.410384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:43.410405 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:43.410535 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:43.410575 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:43.410684 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:43.411168 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 10:56:43.411256 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 10:56:43.413998 27076 log.cc:824] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:43.417768 27076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4: No bootstrap required, opened a new log
I20260928 10:56:43.426350 27076 raft_consensus.cc:348] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead214bbd5ff48d987512c29ced1b5e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36969 } }
I20260928 10:56:43.426837 27076 raft_consensus.cc:374] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:43.426972 27076 raft_consensus.cc:729] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead214bbd5ff48d987512c29ced1b5e4, State: Initialized, Role: FOLLOWER
I20260928 10:56:43.427657 27076 consensus_queue.cc:260] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead214bbd5ff48d987512c29ced1b5e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36969 } }
I20260928 10:56:43.428022 27076 raft_consensus.cc:388] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:43.428192 27076 raft_consensus.cc:482] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:43.428395 27076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:43.431886 27076 raft_consensus.cc:504] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead214bbd5ff48d987512c29ced1b5e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36969 } }
I20260928 10:56:43.432544 27076 leader_election.cc:302] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ead214bbd5ff48d987512c29ced1b5e4; no voters:
I20260928 10:56:43.433584 27076 leader_election.cc:288] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:43.433701 27086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:43.434544 27086 raft_consensus.cc:686] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [term 1 LEADER]: Becoming Leader. State: Replica: ead214bbd5ff48d987512c29ced1b5e4, State: Running, Role: LEADER
I20260928 10:56:43.435225 27086 consensus_queue.cc:237] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead214bbd5ff48d987512c29ced1b5e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36969 } }
I20260928 10:56:43.436129 27076 sys_catalog.cc:564] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:43.438997 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ead214bbd5ff48d987512c29ced1b5e4. Latest consensus state: current_term: 1 leader_uuid: "ead214bbd5ff48d987512c29ced1b5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead214bbd5ff48d987512c29ced1b5e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36969 } } }
I20260928 10:56:43.440261 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:43.438961 27087 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ead214bbd5ff48d987512c29ced1b5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead214bbd5ff48d987512c29ced1b5e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36969 } } }
I20260928 10:56:43.440692 27087 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:43.441980 27093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:43.450104 27093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:43.461946 27093 catalog_manager.cc:1347] Generated new cluster ID: 1fedcd0cdd4743ab973c2daf8de5a284
I20260928 10:56:43.462088 27093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:43.476869 27093 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:43.477397 27093 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:43.485558 27093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ead214bbd5ff48d987512c29ced1b5e4: Generated new TSK 0
I20260928 10:56:43.487115 27093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:43.507294 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27009
W20260928 10:56:43.509294 27071 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58061
W20260928 10:56:43.510280 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:43.510521 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27009
2026-09-28T10:56:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2158 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:43 dist-test-slave-8w8g krb5kdc[27107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:43 dist-test-slave-8w8g krb5kdc[27107](info): set up 2 sockets
Sep 28 10:56:43 dist-test-slave-8w8g krb5kdc[27107](info): commencing operation
krb5kdc: starting...
W20260928 10:56:44.721210 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:44 dist-test-slave-8w8g krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:45.029152 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:45.421416 27120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:45.421839 27120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:45.422006 27120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:45.429708 27120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:45.429906 27120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:45.430027 27120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:45.430135 27120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:45.430210 27120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:45.441329 27120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:45.442894 27120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:45.444458 27120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:45.460222 27128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:45.460263 27125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:45.460287 27126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:45.461853 27120 server_base.cc:1034] running on GCE node
I20260928 10:56:45.462848 27120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:45.464848 27120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:45.466286 27120 hybrid_clock.cc:648] HybridClock initialized: now 1790593005466200 us; error 61 us; skew 500 ppm
I20260928 10:56:45.466842 27120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:45 dist-test-slave-8w8g krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:45.471199 27120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:45.473383 27120 webserver.cc:460] Webserver started at http://127.0.0.1:43557/ using document root <none> and password file <none>
I20260928 10:56:45.474206 27120 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:45.474373 27120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:45.474776 27120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:45.478222 27120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "4c2aa8b289a64444845ceb9c54616316"
format_stamp: "Formatted at 2026-09-28 10:56:45 on dist-test-slave-8w8g"
server_key: "69ed2fae0c79d7c3796fce5905479229"
server_key_iv: "c8df72ee25c173d3e39979fde2bb952e"
server_key_version: "encryptionkey@0"
I20260928 10:56:45.479539 27120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "4c2aa8b289a64444845ceb9c54616316"
format_stamp: "Formatted at 2026-09-28 10:56:45 on dist-test-slave-8w8g"
server_key: "69ed2fae0c79d7c3796fce5905479229"
server_key_iv: "c8df72ee25c173d3e39979fde2bb952e"
server_key_version: "encryptionkey@0"
I20260928 10:56:45.485797 27120 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:56:45.490674 27135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:45.492652 27120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:56:45.493116 27120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "4c2aa8b289a64444845ceb9c54616316"
format_stamp: "Formatted at 2026-09-28 10:56:45 on dist-test-slave-8w8g"
server_key: "69ed2fae0c79d7c3796fce5905479229"
server_key_iv: "c8df72ee25c173d3e39979fde2bb952e"
server_key_version: "encryptionkey@0"
I20260928 10:56:45.493384 27120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:45.509792 27120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:45.515615 27120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:45.516116 27120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:45.541126 27120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41349
I20260928 10:56:45.541177 27186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41349 every 8 connection(s)
I20260928 10:56:45.543121 27120 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:45.544204 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27120
I20260928 10:56:45.544420 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:45.545063 6030 external_mini_cluster.cc:1428] Setting key 43c705842653fde95345e4732f6db803
I20260928 10:56:45.548815 27187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:56:45 dist-test-slave-8w8g krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:56:45.562377 27187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316: Bootstrap starting.
I20260928 10:56:45.567646 27187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:45.569381 27187 log.cc:824] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:45.573958 27187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316: No bootstrap required, opened a new log
I20260928 10:56:45.575194 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:45.547262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:51079 (local address 127.0.0.1:41349)
0928 10:56:45.548234 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:45.548247 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:45.548303 (+ 56us) server_negotiation.cc:407] Connection header received
0928 10:56:45.550754 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:45.550784 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:45.551315 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:45.552010 (+ 695us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:45.553895 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:45.555425 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:45.556886 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:45.557630 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:45.557862 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:45.562939 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:45.562954 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:45.562978 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:45.563068 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:45.568164 (+ 5096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:45.570159 (+ 1995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:45.570174 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:45.570190 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:45.570483 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:45.571155 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:45.571159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:45.571161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:45.571976 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:45.572347 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:45.573288 (+ 941us) server_negotiation.cc:299] Negotiation successful
0928 10:56:45.573702 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":456,"threads_started":1}
I20260928 10:56:45.580873 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:45.585126 27187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aa8b289a64444845ceb9c54616316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260928 10:56:45.585549 27187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:45.585628 27187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2aa8b289a64444845ceb9c54616316, State: Initialized, Role: FOLLOWER
I20260928 10:56:45.586215 27187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aa8b289a64444845ceb9c54616316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260928 10:56:45.586486 27187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:45.586671 27187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:45.586854 27187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:45.589694 27187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aa8b289a64444845ceb9c54616316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260928 10:56:45.590353 27187 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2aa8b289a64444845ceb9c54616316; no voters:
I20260928 10:56:45.591238 27187 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:45.591360 27197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:45.594750 27187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:45.596129 27197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2aa8b289a64444845ceb9c54616316, State: Running, Role: LEADER
I20260928 10:56:45.597296 27197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aa8b289a64444845ceb9c54616316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20260928 10:56:45.605801 27198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2aa8b289a64444845ceb9c54616316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aa8b289a64444845ceb9c54616316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } } }
I20260928 10:56:45.606078 27199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2aa8b289a64444845ceb9c54616316. Latest consensus state: current_term: 1 leader_uuid: "4c2aa8b289a64444845ceb9c54616316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2aa8b289a64444845ceb9c54616316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } } }
I20260928 10:56:45.606657 27198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:45.606734 27199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:45.607831 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:45.587380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:47037 (local address 127.0.0.1:41349)
0928 10:56:45.587716 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:45.587723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:45.587766 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:56:45.588020 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:45.588025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:45.588113 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:45.588237 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:45.589820 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:45.590788 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:45.592961 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:45.593607 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:45.593755 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:45.598163 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:45.598171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:45.598176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:45.598243 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:45.601988 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:45.603087 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:45.603092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:45.603095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:45.603241 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:45.603838 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:45.603846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:45.603851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:45.604321 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:45.604681 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:45.606867 (+ 2186us) server_negotiation.cc:299] Negotiation successful
0928 10:56:45.607223 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:56:45.607896 27206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:45.618090 27206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:45.631369 27206 catalog_manager.cc:1347] Generated new cluster ID: d234821239274e9a85d9447c71ecf173
I20260928 10:56:45.631603 27206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:45.656795 27206 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:45.658318 27206 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:45.665614 27206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2aa8b289a64444845ceb9c54616316: Generated new TSK 0
I20260928 10:56:45.667110 27206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:45.684255 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27120
I20260928 10:56:45.685704 27182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:45.784828 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27120
W20260928 10:56:45.802997 27105 connection.cc:535] client connection to 127.0.0.1:41349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41349): Connection reset by peer (error 104)
2026-09-28T10:56:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:45 dist-test-slave-8w8g krb5kdc[27219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:45 dist-test-slave-8w8g krb5kdc[27219](info): set up 2 sockets
Sep 28 10:56:45 dist-test-slave-8w8g krb5kdc[27219](info): commencing operation
krb5kdc: starting...
W20260928 10:56:47.022290 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:47 dist-test-slave-8w8g krb5kdc[27219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:47.387984 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:47.818429 27232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:47.818812 27232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:47.819001 27232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:47.826846 27232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:47.827049 27232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:47.827175 27232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:47.827286 27232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:47.827368 27232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:47.838003 27232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:47.839466 27232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:47.840967 27232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:47.854918 27237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:47.854882 27240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:47.854982 27238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:47.856565 27232 server_base.cc:1034] running on GCE node
I20260928 10:56:47.857784 27232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:47.859582 27232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:47.860893 27232 hybrid_clock.cc:648] HybridClock initialized: now 1790593007860841 us; error 48 us; skew 500 ppm
I20260928 10:56:47.861344 27232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:47 dist-test-slave-8w8g krb5kdc[27219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:47.871573 27232 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:47.873669 27232 webserver.cc:460] Webserver started at http://127.0.0.1:36965/ using document root <none> and password file <none>
I20260928 10:56:47.874543 27232 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:47.874749 27232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:47.875100 27232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:47.879114 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "e02601af22864e039c9c32067c84cb76"
format_stamp: "Formatted at 2026-09-28 10:56:47 on dist-test-slave-8w8g"
server_key: "ca665be6bd97dce2dbfd366157333169"
server_key_iv: "2238af038a16e53a19438b6dd5bc9a2c"
server_key_version: "encryptionkey@0"
I20260928 10:56:47.880149 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "e02601af22864e039c9c32067c84cb76"
format_stamp: "Formatted at 2026-09-28 10:56:47 on dist-test-slave-8w8g"
server_key: "ca665be6bd97dce2dbfd366157333169"
server_key_iv: "2238af038a16e53a19438b6dd5bc9a2c"
server_key_version: "encryptionkey@0"
I20260928 10:56:47.887004 27232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260928 10:56:47.891659 27247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:47.893427 27232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:56:47.893679 27232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "e02601af22864e039c9c32067c84cb76"
format_stamp: "Formatted at 2026-09-28 10:56:47 on dist-test-slave-8w8g"
server_key: "ca665be6bd97dce2dbfd366157333169"
server_key_iv: "2238af038a16e53a19438b6dd5bc9a2c"
server_key_version: "encryptionkey@0"
I20260928 10:56:47.893936 27232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:47.926865 27232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:47.932677 27232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:47.933046 27232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:47.957496 27298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33495 every 8 connection(s)
I20260928 10:56:47.957566 27232 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33495
I20260928 10:56:47.959210 27232 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:47.961968 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27232
I20260928 10:56:47.962177 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:47.962693 6030 external_mini_cluster.cc:1428] Setting key e04c71cc97bdf6c8f1d71c4b7d191b43
I20260928 10:56:47.964666 27299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:47.978721 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:47.964830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:43931 (local address 127.0.0.1:33495)
0928 10:56:47.965854 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:47.965883 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:47.966216 (+ 333us) server_negotiation.cc:407] Connection header received
0928 10:56:47.968196 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:47.968222 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:47.968866 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:47.969357 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:47.969999 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:47.971459 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:47.973028 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:47.973788 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:47.974010 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:47.975859 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:47.975889 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:47.975916 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:47.976646 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:47.976743 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:47.976778 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:47.976992 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:47.977354 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 10:56:47.977567 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":391,"threads_started":1}
I20260928 10:56:47.979393 27299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76: Bootstrap starting.
I20260928 10:56:47.983589 27299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:47.983995 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:47.984927 27299 log.cc:824] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:47.989041 27299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76: No bootstrap required, opened a new log
I20260928 10:56:47.997910 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:47.988831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:37479 (local address 127.0.0.1:33495)
0928 10:56:47.989084 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:47.989091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:47.989410 (+ 319us) server_negotiation.cc:407] Connection header received
0928 10:56:47.989586 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:47.989591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:47.989688 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:47.989832 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:47.990440 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:47.991638 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:47.993514 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:47.994266 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:47.994416 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:47.996783 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:47.996790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:47.996796 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:47.996823 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:47.996891 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:47.996914 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:47.997066 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:47.997340 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 10:56:47.997418 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:56:48.001225 27299 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02601af22864e039c9c32067c84cb76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260928 10:56:48.001683 27299 raft_consensus.cc:374] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:48.001763 27299 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02601af22864e039c9c32067c84cb76, State: Initialized, Role: FOLLOWER
I20260928 10:56:48.002609 27299 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02601af22864e039c9c32067c84cb76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260928 10:56:48.002889 27299 raft_consensus.cc:388] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:48.003103 27299 raft_consensus.cc:482] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:48.003314 27299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:48.006421 27299 raft_consensus.cc:504] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02601af22864e039c9c32067c84cb76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260928 10:56:48.007089 27299 leader_election.cc:302] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e02601af22864e039c9c32067c84cb76; no voters:
I20260928 10:56:48.008040 27299 leader_election.cc:288] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:48.008216 27309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:48.009093 27309 raft_consensus.cc:686] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [term 1 LEADER]: Becoming Leader. State: Replica: e02601af22864e039c9c32067c84cb76, State: Running, Role: LEADER
I20260928 10:56:48.009924 27309 consensus_queue.cc:237] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02601af22864e039c9c32067c84cb76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260928 10:56:48.010581 27299 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:48.014415 27311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e02601af22864e039c9c32067c84cb76. Latest consensus state: current_term: 1 leader_uuid: "e02601af22864e039c9c32067c84cb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02601af22864e039c9c32067c84cb76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } } }
I20260928 10:56:48.014441 27310 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e02601af22864e039c9c32067c84cb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02601af22864e039c9c32067c84cb76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } } }
I20260928 10:56:48.015058 27310 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:48.015058 27311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:48.016099 27319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:48.023157 27319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:48.034361 27319 catalog_manager.cc:1347] Generated new cluster ID: f9900e8966bc4b3cbb8df23d73f40aa2
I20260928 10:56:48.034497 27319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:48.056391 27319 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:48.058164 27319 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:48.066963 27319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e02601af22864e039c9c32067c84cb76: Generated new TSK 0
I20260928 10:56:48.068141 27319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:48.074761 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27232
W20260928 10:56:48.076280 27294 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:43931
W20260928 10:56:48.077316 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27232: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:48.077529 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27232
W20260928 10:56:48.097075 27217 connection.cc:535] client connection to 127.0.0.1:33495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33495): Connection reset by peer (error 104)
2026-09-28T10:56:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:48 dist-test-slave-8w8g krb5kdc[27330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:48 dist-test-slave-8w8g krb5kdc[27330](info): set up 2 sockets
Sep 28 10:56:48 dist-test-slave-8w8g krb5kdc[27330](info): commencing operation
krb5kdc: starting...
W20260928 10:56:49.256924 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:49 dist-test-slave-8w8g krb5kdc[27330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:49.569437 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:49.945107 27343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:49.945420 27343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:49.945582 27343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:49.954208 27343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:49.954383 27343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:49.954439 27343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:49.954653 27343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:49.954730 27343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:49.966711 27343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:49.968252 27343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:49.970036 27343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:49.983645 27348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:49.983618 27349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:49.983901 27351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:49.985057 27343 server_base.cc:1034] running on GCE node
I20260928 10:56:49.985962 27343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:49.987586 27343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:49.988879 27343 hybrid_clock.cc:648] HybridClock initialized: now 1790593009988801 us; error 96 us; skew 500 ppm
I20260928 10:56:49.989352 27343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:49 dist-test-slave-8w8g krb5kdc[27330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:49.992758 27343 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:49.994392 27343 webserver.cc:460] Webserver started at http://127.0.0.1:32989/ using document root <none> and password file <none>
I20260928 10:56:49.995154 27343 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:49.995277 27343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:49.995566 27343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:49.998999 27343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "49c305a4c2304232a8a47407e231b059"
format_stamp: "Formatted at 2026-09-28 10:56:49 on dist-test-slave-8w8g"
server_key: "63b707608959d67d7e157d061bec913c"
server_key_iv: "a07c3332710fce33b021c06a00e52429"
server_key_version: "encryptionkey@0"
I20260928 10:56:50.000213 27343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "49c305a4c2304232a8a47407e231b059"
format_stamp: "Formatted at 2026-09-28 10:56:49 on dist-test-slave-8w8g"
server_key: "63b707608959d67d7e157d061bec913c"
server_key_iv: "a07c3332710fce33b021c06a00e52429"
server_key_version: "encryptionkey@0"
I20260928 10:56:50.006824 27343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 10:56:50.011204 27358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:50.013193 27343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:50.013429 27343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "49c305a4c2304232a8a47407e231b059"
format_stamp: "Formatted at 2026-09-28 10:56:49 on dist-test-slave-8w8g"
server_key: "63b707608959d67d7e157d061bec913c"
server_key_iv: "a07c3332710fce33b021c06a00e52429"
server_key_version: "encryptionkey@0"
I20260928 10:56:50.013682 27343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:50.021240 27343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:50.026292 27343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:50.026669 27343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:50.048667 27343 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41823
I20260928 10:56:50.048662 27409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41823 every 8 connection(s)
I20260928 10:56:50.050366 27343 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:50.051534 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27343
I20260928 10:56:50.051790 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:50.052395 6030 external_mini_cluster.cc:1428] Setting key 499d2d4aa373fc57543f572c31c6bb16
I20260928 10:56:50.055963 27410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:50.062279 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:50.054520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:35959 (local address 127.0.0.1:41823)
0928 10:56:50.055418 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:50.055436 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:50.055493 (+ 57us) server_negotiation.cc:407] Connection header received
0928 10:56:50.057945 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:50.057976 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:50.058460 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:50.058964 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:50.060113 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:50.060122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:50.060140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:50.060164 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:50.060249 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:50.060276 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:50.060514 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:50.060673 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 10:56:50.060907 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":350,"threads_started":1}
I20260928 10:56:50.066742 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:50.070982 27410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059: Bootstrap starting.
I20260928 10:56:50.075085 27410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:50.075929 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:50.072715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:34929 (local address 127.0.0.1:41823)
0928 10:56:50.072983 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:50.072992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:50.073502 (+ 510us) server_negotiation.cc:407] Connection header received
0928 10:56:50.073662 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:50.073668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:50.073731 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:50.073854 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:50.074957 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:50.074965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:50.074969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:50.074990 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:50.075041 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:50.075073 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:50.075201 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:50.075378 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 10:56:50.075556 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 10:56:50.076611 27410 log.cc:824] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:50.079869 27410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059: No bootstrap required, opened a new log
I20260928 10:56:50.087863 27410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c305a4c2304232a8a47407e231b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260928 10:56:50.088351 27410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:50.088469 27410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c305a4c2304232a8a47407e231b059, State: Initialized, Role: FOLLOWER
I20260928 10:56:50.089121 27410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c305a4c2304232a8a47407e231b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260928 10:56:50.089396 27410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:50.089553 27410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:50.089735 27410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:50.092332 27410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c305a4c2304232a8a47407e231b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260928 10:56:50.092953 27410 leader_election.cc:302] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c305a4c2304232a8a47407e231b059; no voters:
I20260928 10:56:50.093645 27410 leader_election.cc:288] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:50.094007 27420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:50.094836 27420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [term 1 LEADER]: Becoming Leader. State: Replica: 49c305a4c2304232a8a47407e231b059, State: Running, Role: LEADER
I20260928 10:56:50.095472 27420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c305a4c2304232a8a47407e231b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260928 10:56:50.096050 27410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:50.098656 27421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49c305a4c2304232a8a47407e231b059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c305a4c2304232a8a47407e231b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } } }
I20260928 10:56:50.099282 27421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:50.098682 27422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c305a4c2304232a8a47407e231b059. Latest consensus state: current_term: 1 leader_uuid: "49c305a4c2304232a8a47407e231b059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c305a4c2304232a8a47407e231b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } } }
I20260928 10:56:50.099514 27422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:50.103574 27426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:50.111033 27426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:50.122485 27426 catalog_manager.cc:1347] Generated new cluster ID: 357745bf89cb40a0ac9af150a2b15eea
I20260928 10:56:50.122629 27426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:50.145843 27426 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:50.146404 27426 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:50.158207 27426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49c305a4c2304232a8a47407e231b059: Generated new TSK 0
I20260928 10:56:50.159195 27426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:50.173063 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27343
W20260928 10:56:50.175257 27405 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:35959
W20260928 10:56:50.176183 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:50.176374 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27343
2026-09-28T10:56:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2098 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:50 dist-test-slave-8w8g krb5kdc[27441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:50 dist-test-slave-8w8g krb5kdc[27441](info): set up 2 sockets
Sep 28 10:56:50 dist-test-slave-8w8g krb5kdc[27441](info): commencing operation
krb5kdc: starting...
W20260928 10:56:51.341034 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:51 dist-test-slave-8w8g krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:51.649412 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:52.019048 27454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:52.019472 27454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:52.019652 27454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:52.027335 27454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:52.027529 27454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:52.027650 27454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:52.027786 27454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:52.027863 27454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:52.038467 27454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:52.040030 27454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:52.041441 27454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:52.055881 27462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:52.055812 27459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:52.055845 27460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:52.057632 27454 server_base.cc:1034] running on GCE node
I20260928 10:56:52.058717 27454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:52.060559 27454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:52.061849 27454 hybrid_clock.cc:648] HybridClock initialized: now 1790593012061814 us; error 55 us; skew 500 ppm
I20260928 10:56:52.062305 27454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:52 dist-test-slave-8w8g krb5kdc[27441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:52.065996 27454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:52.067813 27454 webserver.cc:460] Webserver started at http://127.0.0.1:43513/ using document root <none> and password file <none>
I20260928 10:56:52.068593 27454 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:52.068747 27454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:52.069094 27454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:52.072697 27454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "b7e93114b9b84f9480453f68d95cd0a8"
format_stamp: "Formatted at 2026-09-28 10:56:52 on dist-test-slave-8w8g"
server_key: "8f9dd04848a8f8863bf691e76ed0c118"
server_key_iv: "581fddfa190929002ee602ca23b1c4b7"
server_key_version: "encryptionkey@0"
I20260928 10:56:52.073701 27454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "b7e93114b9b84f9480453f68d95cd0a8"
format_stamp: "Formatted at 2026-09-28 10:56:52 on dist-test-slave-8w8g"
server_key: "8f9dd04848a8f8863bf691e76ed0c118"
server_key_iv: "581fddfa190929002ee602ca23b1c4b7"
server_key_version: "encryptionkey@0"
I20260928 10:56:52.079715 27454 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 10:56:52.083909 27469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:52.085748 27454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:56:52.086017 27454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "b7e93114b9b84f9480453f68d95cd0a8"
format_stamp: "Formatted at 2026-09-28 10:56:52 on dist-test-slave-8w8g"
server_key: "8f9dd04848a8f8863bf691e76ed0c118"
server_key_iv: "581fddfa190929002ee602ca23b1c4b7"
server_key_version: "encryptionkey@0"
I20260928 10:56:52.086297 27454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:52.100420 27454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:52.105394 27454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:52.105719 27454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:52.127487 27454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36845
I20260928 10:56:52.127552 27520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36845 every 8 connection(s)
I20260928 10:56:52.129082 27454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:52.129619 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27454
I20260928 10:56:52.129801 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:52.130470 6030 external_mini_cluster.cc:1428] Setting key a5b7fa626282d2ac11dcbbcd44faeb32
I20260928 10:56:52.134848 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:56:52 dist-test-slave-8w8g krb5kdc[27441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:56:52.148272 27521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8: Bootstrap starting.
I20260928 10:56:52.153153 27521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:52.155478 27521 log.cc:824] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:52.158686 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:52.132764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:41679 (local address 127.0.0.1:36845)
0928 10:56:52.133824 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:52.133838 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:52.133884 (+ 46us) server_negotiation.cc:407] Connection header received
0928 10:56:52.136241 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:52.136267 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:52.136725 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:52.137219 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:52.138848 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:52.140363 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:52.141734 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:52.142487 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:52.142706 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:52.147452 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:52.147462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:52.147480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:52.147529 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:52.151477 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:52.152782 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:52.152793 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:52.152800 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:52.153102 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:52.154048 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:52.154057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:52.154061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:52.155108 (+ 1047us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:52.155517 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:52.156841 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0928 10:56:52.157171 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":471,"threads_started":1}
I20260928 10:56:52.159652 27521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8: No bootstrap required, opened a new log
I20260928 10:56:52.164157 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:52.171084 27521 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e93114b9b84f9480453f68d95cd0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 10:56:52.171629 27521 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:52.171780 27521 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e93114b9b84f9480453f68d95cd0a8, State: Initialized, Role: FOLLOWER
I20260928 10:56:52.172429 27521 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e93114b9b84f9480453f68d95cd0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 10:56:52.172652 27521 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:52.172792 27521 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:52.172992 27521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:52.176067 27521 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e93114b9b84f9480453f68d95cd0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 10:56:52.176795 27521 leader_election.cc:302] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e93114b9b84f9480453f68d95cd0a8; no voters:
I20260928 10:56:52.177573 27521 leader_election.cc:288] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:52.177758 27531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:52.180413 27521 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:52.182042 27531 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [term 1 LEADER]: Becoming Leader. State: Replica: b7e93114b9b84f9480453f68d95cd0a8, State: Running, Role: LEADER
I20260928 10:56:52.182811 27531 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e93114b9b84f9480453f68d95cd0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260928 10:56:52.186784 27533 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e93114b9b84f9480453f68d95cd0a8. Latest consensus state: current_term: 1 leader_uuid: "b7e93114b9b84f9480453f68d95cd0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e93114b9b84f9480453f68d95cd0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260928 10:56:52.187235 27533 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:52.187929 27532 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7e93114b9b84f9480453f68d95cd0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e93114b9b84f9480453f68d95cd0a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260928 10:56:52.188277 27532 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:52.193643 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:52.170050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:49053 (local address 127.0.0.1:36845)
0928 10:56:52.170243 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:52.170250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:52.170808 (+ 558us) server_negotiation.cc:407] Connection header received
0928 10:56:52.170916 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:52.170923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:52.171067 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:52.171188 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:52.173403 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:52.174662 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:52.176288 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:52.176991 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:52.177136 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:52.181898 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:52.181910 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:52.181919 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:52.182005 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:52.187276 (+ 5271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:52.188746 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:52.188756 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:52.188762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:52.189007 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:52.189809 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:52.189817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:52.189823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:52.190464 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:52.192482 (+ 2018us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:52.192733 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 10:56:52.193030 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:56:52.197551 27539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:52.204000 27539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:52.216034 27539 catalog_manager.cc:1347] Generated new cluster ID: 712e67fa7e014d9a975f52a60ebc3d8e
I20260928 10:56:52.216178 27539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:52.230669 27539 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:52.232226 27539 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:52.242450 27539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7e93114b9b84f9480453f68d95cd0a8: Generated new TSK 0
I20260928 10:56:52.243855 27539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:52.255162 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27454
I20260928 10:56:52.256543 27516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:52.349140 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27454
W20260928 10:56:52.367794 27439 connection.cc:535] client connection to 127.0.0.1:36845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36845): Connection reset by peer (error 104)
2026-09-28T10:56:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2171 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:52 dist-test-slave-8w8g krb5kdc[27553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:52 dist-test-slave-8w8g krb5kdc[27553](info): set up 2 sockets
Sep 28 10:56:52 dist-test-slave-8w8g krb5kdc[27553](info): commencing operation
krb5kdc: starting...
W20260928 10:56:53.513355 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:53 dist-test-slave-8w8g krb5kdc[27553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:53.846825 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:54.272532 27566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:54.272877 27566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:54.272982 27566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:54.282392 27566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:54.282564 27566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:54.282615 27566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:54.282670 27566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:54.282716 27566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:54.295152 27566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:54.296613 27566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:54.298194 27566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:54.314339 27572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:54.314312 27571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:54.315604 27574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:54.316128 27566 server_base.cc:1034] running on GCE node
I20260928 10:56:54.317065 27566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:54.318878 27566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:54.320233 27566 hybrid_clock.cc:648] HybridClock initialized: now 1790593014320155 us; error 66 us; skew 500 ppm
I20260928 10:56:54.320783 27566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:54 dist-test-slave-8w8g krb5kdc[27553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:54.330818 27566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:54.332849 27566 webserver.cc:460] Webserver started at http://127.0.0.1:40905/ using document root <none> and password file <none>
I20260928 10:56:54.333638 27566 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:54.333873 27566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:54.334220 27566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:54.338620 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "461868b8f7b844b3b1a70cb15ef6b163"
format_stamp: "Formatted at 2026-09-28 10:56:54 on dist-test-slave-8w8g"
server_key: "66ed483de727961e1486eeb2d59dadc1"
server_key_iv: "ee5e9af4217d2040869b7e1fecfac74c"
server_key_version: "encryptionkey@0"
I20260928 10:56:54.339635 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "461868b8f7b844b3b1a70cb15ef6b163"
format_stamp: "Formatted at 2026-09-28 10:56:54 on dist-test-slave-8w8g"
server_key: "66ed483de727961e1486eeb2d59dadc1"
server_key_iv: "ee5e9af4217d2040869b7e1fecfac74c"
server_key_version: "encryptionkey@0"
I20260928 10:56:54.347622 27566 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:56:54.352945 27581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:54.355284 27566 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260928 10:56:54.355497 27566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "461868b8f7b844b3b1a70cb15ef6b163"
format_stamp: "Formatted at 2026-09-28 10:56:54 on dist-test-slave-8w8g"
server_key: "66ed483de727961e1486eeb2d59dadc1"
server_key_iv: "ee5e9af4217d2040869b7e1fecfac74c"
server_key_version: "encryptionkey@0"
I20260928 10:56:54.355834 27566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:54.372645 27566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:54.378690 27566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:54.379137 27566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:54.403240 27566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33739
I20260928 10:56:54.403273 27632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33739 every 8 connection(s)
I20260928 10:56:54.404784 27566 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:54.409171 27633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:54.410177 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27566
I20260928 10:56:54.410576 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:54.411329 6030 external_mini_cluster.cc:1428] Setting key 4cc76217cd0dbc343eacc498ffb787eb
I20260928 10:56:54.426290 27633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163: Bootstrap starting.
I20260928 10:56:54.429685 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:54.414252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:40139 (local address 127.0.0.1:33739)
0928 10:56:54.415353 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:54.415370 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:54.415423 (+ 53us) server_negotiation.cc:407] Connection header received
0928 10:56:54.417314 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:54.417348 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:54.417888 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:54.418302 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:54.419102 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:54.420843 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:54.423202 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:54.424278 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:54.424500 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:54.426365 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:54.426427 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:54.426459 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:54.427210 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:54.427345 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:54.427376 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:54.427645 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:54.428145 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 10:56:54.428399 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":540,"threads_started":1}
I20260928 10:56:54.430697 27633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:54.432135 27633 log.cc:824] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:54.436079 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:54.436743 27633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163: No bootstrap required, opened a new log
I20260928 10:56:54.447064 27633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461868b8f7b844b3b1a70cb15ef6b163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260928 10:56:54.447580 27633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:54.447718 27633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461868b8f7b844b3b1a70cb15ef6b163, State: Initialized, Role: FOLLOWER
I20260928 10:56:54.448671 27633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461868b8f7b844b3b1a70cb15ef6b163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260928 10:56:54.449043 27633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:54.449230 27633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:54.449659 27633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:54.450069 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:54.441940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:40983 (local address 127.0.0.1:33739)
0928 10:56:54.442210 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:54.442219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:54.442677 (+ 458us) server_negotiation.cc:407] Connection header received
0928 10:56:54.442957 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:54.443193 (+ 236us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:54.443300 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:54.443499 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:54.444343 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:54.445308 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:54.446737 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:54.447200 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:54.447351 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:54.448828 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:54.448834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:54.448837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:54.448860 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:54.448911 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:54.448928 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:54.449122 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:54.449544 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 10:56:54.449604 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:56:54.454223 27633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461868b8f7b844b3b1a70cb15ef6b163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260928 10:56:54.455744 27633 leader_election.cc:302] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 461868b8f7b844b3b1a70cb15ef6b163; no voters:
I20260928 10:56:54.456957 27633 leader_election.cc:288] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:54.457094 27643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:54.458065 27643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [term 1 LEADER]: Becoming Leader. State: Replica: 461868b8f7b844b3b1a70cb15ef6b163, State: Running, Role: LEADER
I20260928 10:56:54.458804 27643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461868b8f7b844b3b1a70cb15ef6b163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } }
I20260928 10:56:54.462191 27633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:54.463084 27645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "461868b8f7b844b3b1a70cb15ef6b163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461868b8f7b844b3b1a70cb15ef6b163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } } }
I20260928 10:56:54.462710 27644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 461868b8f7b844b3b1a70cb15ef6b163. Latest consensus state: current_term: 1 leader_uuid: "461868b8f7b844b3b1a70cb15ef6b163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461868b8f7b844b3b1a70cb15ef6b163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33739 } } }
I20260928 10:56:54.463569 27645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:54.463683 27644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:54.465029 27647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:54.473520 27647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:54.485438 27647 catalog_manager.cc:1347] Generated new cluster ID: ede2841b3ea1446d8b00139786092498
I20260928 10:56:54.485615 27647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:54.505206 27647 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:54.506848 27647 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:54.519958 27647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 461868b8f7b844b3b1a70cb15ef6b163: Generated new TSK 0
I20260928 10:56:54.521087 27647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:54.532289 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27566
W20260928 10:56:54.534824 27628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:40139
W20260928 10:56:54.536348 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:54.536808 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27566
W20260928 10:56:54.556982 27551 connection.cc:535] client connection to 127.0.0.1:33739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33739): Connection reset by peer (error 104)
2026-09-28T10:56:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2189 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:54 dist-test-slave-8w8g krb5kdc[27664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:54 dist-test-slave-8w8g krb5kdc[27664](info): set up 2 sockets
Sep 28 10:56:54 dist-test-slave-8w8g krb5kdc[27664](info): commencing operation
krb5kdc: starting...
W20260928 10:56:55.785494 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:55 dist-test-slave-8w8g krb5kdc[27664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:56:56.108021 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:56.514873 27677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:56.515213 27677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:56.515406 27677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:56.523439 27677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:56.523604 27677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:56.523661 27677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:56.523821 27677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:56.523873 27677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:56.538743 27677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:56.540365 27677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:56.542289 27677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:56.558985 27682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:56.559010 27683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:56.559000 27685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:56.562112 27677 server_base.cc:1034] running on GCE node
I20260928 10:56:56.563294 27677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:56.565487 27677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:56.566900 27677 hybrid_clock.cc:648] HybridClock initialized: now 1790593016566860 us; error 65 us; skew 500 ppm
I20260928 10:56:56.567507 27677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:56 dist-test-slave-8w8g krb5kdc[27664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:56.573429 27677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:56:56.576563 27677 webserver.cc:460] Webserver started at http://127.0.0.1:41673/ using document root <none> and password file <none>
I20260928 10:56:56.577555 27677 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:56.578084 27677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:56.578863 27677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:56.584244 27677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "f1f3fbbc068b473280936c12c8b08cb1"
format_stamp: "Formatted at 2026-09-28 10:56:56 on dist-test-slave-8w8g"
server_key: "91ab0499efc32ba9d3be61e2077a6fa5"
server_key_iv: "99d32cbe7453fda406355b4a8d669989"
server_key_version: "encryptionkey@0"
I20260928 10:56:56.585701 27677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "f1f3fbbc068b473280936c12c8b08cb1"
format_stamp: "Formatted at 2026-09-28 10:56:56 on dist-test-slave-8w8g"
server_key: "91ab0499efc32ba9d3be61e2077a6fa5"
server_key_iv: "99d32cbe7453fda406355b4a8d669989"
server_key_version: "encryptionkey@0"
I20260928 10:56:56.595134 27677 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260928 10:56:56.601136 27692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:56.603519 27677 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260928 10:56:56.603801 27677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "f1f3fbbc068b473280936c12c8b08cb1"
format_stamp: "Formatted at 2026-09-28 10:56:56 on dist-test-slave-8w8g"
server_key: "91ab0499efc32ba9d3be61e2077a6fa5"
server_key_iv: "99d32cbe7453fda406355b4a8d669989"
server_key_version: "encryptionkey@0"
I20260928 10:56:56.604079 27677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:56.611332 27677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:56.617233 27677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:56.617606 27677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:56.642657 27677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33941
I20260928 10:56:56.643165 27743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33941 every 8 connection(s)
I20260928 10:56:56.644515 27677 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:56.649467 27744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:56.650704 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27677
I20260928 10:56:56.650954 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:56.651454 6030 external_mini_cluster.cc:1428] Setting key bb812eb3c5e90183f9944bc82d50458f
I20260928 10:56:56.663415 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:56.653961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:35679 (local address 127.0.0.1:33941)
0928 10:56:56.655297 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:56.655318 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:56.656127 (+ 809us) server_negotiation.cc:407] Connection header received
0928 10:56:56.658814 (+ 2687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:56.658848 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:56.659396 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:56.659985 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:56.661100 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:56.661108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:56.661129 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:56.661147 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:56.661227 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:56.661256 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:56.661519 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:56.661708 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 10:56:56.661943 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":666,"threads_started":1}
I20260928 10:56:56.665170 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1: Bootstrap starting.
I20260928 10:56:56.668998 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:56.669548 27744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:56.671074 27744 log.cc:824] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:56.675597 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1: No bootstrap required, opened a new log
I20260928 10:56:56.678922 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:56.675933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:59639 (local address 127.0.0.1:33941)
0928 10:56:56.676106 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:56.676110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:56.676578 (+ 468us) server_negotiation.cc:407] Connection header received
0928 10:56:56.676761 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:56.676768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:56.676843 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:56.676967 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:56.678099 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:56.678107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:56.678113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:56:56.678141 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:56.678200 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:56:56.678227 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:56.678365 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:56.678515 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 10:56:56.678598 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 10:56:56.685428 27744 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f3fbbc068b473280936c12c8b08cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260928 10:56:56.686040 27744 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:56.686167 27744 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f3fbbc068b473280936c12c8b08cb1, State: Initialized, Role: FOLLOWER
I20260928 10:56:56.686925 27744 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f3fbbc068b473280936c12c8b08cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260928 10:56:56.687170 27744 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:56.687269 27744 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:56.687397 27744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:56.690228 27744 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f3fbbc068b473280936c12c8b08cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260928 10:56:56.690948 27744 leader_election.cc:302] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1f3fbbc068b473280936c12c8b08cb1; no voters:
I20260928 10:56:56.691833 27744 leader_election.cc:288] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:56.692036 27754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:56.692771 27754 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [term 1 LEADER]: Becoming Leader. State: Replica: f1f3fbbc068b473280936c12c8b08cb1, State: Running, Role: LEADER
I20260928 10:56:56.693387 27754 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f3fbbc068b473280936c12c8b08cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260928 10:56:56.694255 27744 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:56.696513 27756 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1f3fbbc068b473280936c12c8b08cb1. Latest consensus state: current_term: 1 leader_uuid: "f1f3fbbc068b473280936c12c8b08cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f3fbbc068b473280936c12c8b08cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260928 10:56:56.696684 27755 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1f3fbbc068b473280936c12c8b08cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f3fbbc068b473280936c12c8b08cb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260928 10:56:56.697005 27756 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:56.697085 27755 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:56.700960 27761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:56.708659 27761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:56.720222 27761 catalog_manager.cc:1347] Generated new cluster ID: 8703d4a09eff46f4989a6aa3fd56321b
I20260928 10:56:56.720381 27761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:56.734583 27761 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:56.735170 27761 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:56.747259 27761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1f3fbbc068b473280936c12c8b08cb1: Generated new TSK 0
I20260928 10:56:56.748565 27761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:56.762396 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27677
W20260928 10:56:56.763978 27739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:35679
W20260928 10:56:56.764853 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27677: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:56:56.765046 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27677
2026-09-28T10:56:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:56 dist-test-slave-8w8g krb5kdc[27775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:56 dist-test-slave-8w8g krb5kdc[27775](info): set up 2 sockets
Sep 28 10:56:56 dist-test-slave-8w8g krb5kdc[27775](info): commencing operation
krb5kdc: starting...
W20260928 10:56:57.997334 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:56:58 dist-test-slave-8w8g krb5kdc[27775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:56:58.321017 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:33351
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:33351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:56:58.716357 27788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:56:58.716699 27788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:56:58.716881 27788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:56:58.726094 27788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:56:58.726254 27788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:56:58.726307 27788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:56:58.726657 27788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:56:58.726902 27788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:56:58.738907 27788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:33351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:33351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:56:58.740691 27788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:56:58.742602 27788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:56:58.759048 27796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:58.759016 27794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:56:58.759986 27793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:56:58.760129 27788 server_base.cc:1034] running on GCE node
I20260928 10:56:58.761217 27788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:56:58.762935 27788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:56:58.766574 27788 hybrid_clock.cc:648] HybridClock initialized: now 1790593018766453 us; error 81 us; skew 500 ppm
I20260928 10:56:58.767148 27788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:56:58 dist-test-slave-8w8g krb5kdc[27775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593018, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:56:58.771250 27788 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:56:58.773430 27788 webserver.cc:460] Webserver started at http://127.5.227.190:39619/ using document root <none> and password file <none>
I20260928 10:56:58.774251 27788 fs_manager.cc:359] Metadata directory not provided
I20260928 10:56:58.774430 27788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:56:58.774881 27788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:56:58.778329 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "dd4a04dcbb9c46538476d505a69956a0"
format_stamp: "Formatted at 2026-09-28 10:56:58 on dist-test-slave-8w8g"
server_key: "ab3c93da7ba16723834e78a47fe6f4b6"
server_key_iv: "cc0b30f02d3581ab340244a2f8818602"
server_key_version: "encryptionkey@0"
I20260928 10:56:58.779224 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "dd4a04dcbb9c46538476d505a69956a0"
format_stamp: "Formatted at 2026-09-28 10:56:58 on dist-test-slave-8w8g"
server_key: "ab3c93da7ba16723834e78a47fe6f4b6"
server_key_iv: "cc0b30f02d3581ab340244a2f8818602"
server_key_version: "encryptionkey@0"
I20260928 10:56:58.785331 27788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:56:58.789655 27803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:56:58.791817 27788 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:56:58.792104 27788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "dd4a04dcbb9c46538476d505a69956a0"
format_stamp: "Formatted at 2026-09-28 10:56:58 on dist-test-slave-8w8g"
server_key: "ab3c93da7ba16723834e78a47fe6f4b6"
server_key_iv: "cc0b30f02d3581ab340244a2f8818602"
server_key_version: "encryptionkey@0"
I20260928 10:56:58.792348 27788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:56:58.816423 27788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:56:58.822358 27788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:56:58.822695 27788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:56:58.845170 27788 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:33351
I20260928 10:56:58.845207 27854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:33351 every 8 connection(s)
I20260928 10:56:58.846714 27788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:56:58.851130 27855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:56:58.852784 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27788
I20260928 10:56:58.853019 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:56:58.853664 6030 external_mini_cluster.cc:1428] Setting key 8116b9f0518b4d09a964528e55ccde9c
I20260928 10:56:58.866055 27855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0: Bootstrap starting.
Sep 28 10:56:58 dist-test-slave-8w8g krb5kdc[27775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:56:58.870126 27855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0: Neither blocks nor log segments found. Creating new log.
I20260928 10:56:58.871533 27855 log.cc:824] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:56:58.876525 27855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0: No bootstrap required, opened a new log
I20260928 10:56:58.882292 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:58.856042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60897 (local address 127.5.227.190:33351)
0928 10:56:58.857065 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:58.857080 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:58.857132 (+ 52us) server_negotiation.cc:407] Connection header received
0928 10:56:58.858902 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:58.858946 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:58.859475 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:58.860033 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:58.862064 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:58.863491 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:58.864988 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:58.865743 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:58.865971 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:58.871327 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:58.871340 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:58.871362 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:58.871417 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:58.876380 (+ 4963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:58.877571 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:58.877582 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:58.877591 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:58.877834 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:58.878442 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:58.878447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:58.878450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:58.879270 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:58.879829 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:58.880310 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 10:56:58.880738 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":523,"threads_started":1}
I20260928 10:56:58.887234 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:56:58.887559 27855 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4a04dcbb9c46538476d505a69956a0" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33351 } }
I20260928 10:56:58.888041 27855 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:56:58.888161 27855 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4a04dcbb9c46538476d505a69956a0, State: Initialized, Role: FOLLOWER
I20260928 10:56:58.888827 27855 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4a04dcbb9c46538476d505a69956a0" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33351 } }
I20260928 10:56:58.889091 27855 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:56:58.889242 27855 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:56:58.889415 27855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:56:58.892946 27855 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4a04dcbb9c46538476d505a69956a0" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33351 } }
I20260928 10:56:58.893718 27855 leader_election.cc:302] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd4a04dcbb9c46538476d505a69956a0; no voters:
I20260928 10:56:58.894761 27855 leader_election.cc:288] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:56:58.894806 27865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:56:58.896618 27865 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [term 1 LEADER]: Becoming Leader. State: Replica: dd4a04dcbb9c46538476d505a69956a0, State: Running, Role: LEADER
I20260928 10:56:58.897320 27865 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4a04dcbb9c46538476d505a69956a0" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33351 } }
I20260928 10:56:58.897866 27855 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:56:58.900314 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd4a04dcbb9c46538476d505a69956a0. Latest consensus state: current_term: 1 leader_uuid: "dd4a04dcbb9c46538476d505a69956a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4a04dcbb9c46538476d505a69956a0" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33351 } } }
I20260928 10:56:58.900898 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:58.900914 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd4a04dcbb9c46538476d505a69956a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4a04dcbb9c46538476d505a69956a0" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 33351 } } }
I20260928 10:56:58.902623 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:56:58.915939 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:56:58.892847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60147 (local address 127.5.227.190:33351)
0928 10:56:58.893085 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 10:56:58.893091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:56:58.893462 (+ 371us) server_negotiation.cc:407] Connection header received
0928 10:56:58.893687 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:56:58.893692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:56:58.893770 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:56:58.893870 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:56:58.896499 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:58.897971 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:58.900462 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:56:58.901248 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:56:58.901445 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:56:58.906179 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:56:58.906191 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:56:58.906199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:56:58.906266 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:56:58.912009 (+ 5743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:58.913229 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:58.913235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:58.913239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:58.913397 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:56:58.913996 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:56:58.914008 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:56:58.914011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:56:58.914394 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:56:58.914729 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 10:56:58.915225 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 10:56:58.915463 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 10:56:58.920575 27870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:56:58.928610 27870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:56:58.940832 27870 catalog_manager.cc:1347] Generated new cluster ID: 710c4a07da294bda954cf003d87b0752
I20260928 10:56:58.940982 27870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:56:58.956043 27870 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:56:58.957710 27870 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:56:58.972070 27870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd4a04dcbb9c46538476d505a69956a0: Generated new TSK 0
I20260928 10:56:58.973122 27870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:56:58.992143 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27788
I20260928 10:56:58.993731 27850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:56:59.095719 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27788
W20260928 10:56:59.118050 27773 connection.cc:535] client connection to 127.5.227.190:33351 recv error: Network error: failed to read from TLS socket (remote: 127.5.227.190:33351): Connection reset by peer (error 104)
2026-09-28T10:56:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:56:59 dist-test-slave-8w8g krb5kdc[27887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:56:59 dist-test-slave-8w8g krb5kdc[27887](info): set up 2 sockets
Sep 28 10:56:59 dist-test-slave-8w8g krb5kdc[27887](info): commencing operation
krb5kdc: starting...
W20260928 10:57:00.285465 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:57:00 dist-test-slave-8w8g krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:57:00.606570 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39987
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:45219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:00.993438 27900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:00.993770 27900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:00.993958 27900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:01.002615 27900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:01.002786 27900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:01.002907 27900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:01.003019 27900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:01.003072 27900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:01.015026 27900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:45219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:01.016640 27900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:01.018116 27900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:01.032862 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:01.033253 27900 server_base.cc:1034] running on GCE node
W20260928 10:57:01.032984 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:01.033032 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:01.034586 27900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:01.037035 27900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:01.038385 27900 hybrid_clock.cc:648] HybridClock initialized: now 1790593021038292 us; error 89 us; skew 500 ppm
I20260928 10:57:01.038878 27900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:01 dist-test-slave-8w8g krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593021, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:01.050519 27900 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:57:01.052554 27900 webserver.cc:460] Webserver started at http://127.5.227.190:33827/ using document root <none> and password file <none>
I20260928 10:57:01.053344 27900 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:01.053699 27900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:01.054104 27900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:01.057950 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "a2267a4b7199487cacd65d135ff1fe99"
format_stamp: "Formatted at 2026-09-28 10:57:01 on dist-test-slave-8w8g"
server_key: "e65f7226bf97c218a39dd3a430fb6029"
server_key_iv: "894fdc7ee0f1e99ea0b59bd96f1200be"
server_key_version: "encryptionkey@0"
I20260928 10:57:01.058974 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "a2267a4b7199487cacd65d135ff1fe99"
format_stamp: "Formatted at 2026-09-28 10:57:01 on dist-test-slave-8w8g"
server_key: "e65f7226bf97c218a39dd3a430fb6029"
server_key_iv: "894fdc7ee0f1e99ea0b59bd96f1200be"
server_key_version: "encryptionkey@0"
I20260928 10:57:01.065603 27900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260928 10:57:01.070384 27915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:01.072270 27900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:57:01.072515 27900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "a2267a4b7199487cacd65d135ff1fe99"
format_stamp: "Formatted at 2026-09-28 10:57:01 on dist-test-slave-8w8g"
server_key: "e65f7226bf97c218a39dd3a430fb6029"
server_key_iv: "894fdc7ee0f1e99ea0b59bd96f1200be"
server_key_version: "encryptionkey@0"
I20260928 10:57:01.072753 27900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:01.147100 27900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:01.152357 27900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:01.152693 27900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:01.177541 27900 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:39987
I20260928 10:57:01.177940 27966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:39987 every 8 connection(s)
I20260928 10:57:01.179898 27900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:01.185278 27967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:01.189821 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 27900
I20260928 10:57:01.190261 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:57:01.190974 6030 external_mini_cluster.cc:1428] Setting key cc75580c95bde83289b7f98e1ad14a03
I20260928 10:57:01.198657 27967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99: Bootstrap starting.
I20260928 10:57:01.203186 27967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:01.205128 27967 log.cc:824] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:01.210613 27967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99: No bootstrap required, opened a new log
I20260928 10:57:01.212153 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:01.193910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33023 (local address 127.5.227.190:39987)
0928 10:57:01.195108 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:01.195121 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:01.195170 (+ 49us) server_negotiation.cc:407] Connection header received
0928 10:57:01.197458 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:01.197494 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:01.198010 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:01.198420 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:01.199128 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:01.201036 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:01.203710 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:01.205102 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:01.205369 (+ 267us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:01.207947 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:01.207960 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:01.208059 (+ 99us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:01.208944 (+ 885us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:01.209103 (+ 159us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:01.209150 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:01.209356 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:01.209890 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 10:57:01.210372 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":610,"threads_started":1}
I20260928 10:57:01.217643 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:57:01.221575 27967 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2267a4b7199487cacd65d135ff1fe99" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39987 } }
I20260928 10:57:01.222249 27967 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:01.222421 27967 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2267a4b7199487cacd65d135ff1fe99, State: Initialized, Role: FOLLOWER
I20260928 10:57:01.223210 27967 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2267a4b7199487cacd65d135ff1fe99" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39987 } }
I20260928 10:57:01.223515 27967 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:01.223762 27967 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:01.224014 27967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:01.227337 27967 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2267a4b7199487cacd65d135ff1fe99" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39987 } }
I20260928 10:57:01.228106 27967 leader_election.cc:302] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2267a4b7199487cacd65d135ff1fe99; no voters:
I20260928 10:57:01.228886 27967 leader_election.cc:288] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:01.229036 27977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:01.230340 27977 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [term 1 LEADER]: Becoming Leader. State: Replica: a2267a4b7199487cacd65d135ff1fe99, State: Running, Role: LEADER
I20260928 10:57:01.231030 27977 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2267a4b7199487cacd65d135ff1fe99" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39987 } }
I20260928 10:57:01.231652 27967 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:01.233144 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:01.224114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55779 (local address 127.5.227.190:39987)
0928 10:57:01.224326 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:01.224338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:01.224969 (+ 631us) server_negotiation.cc:407] Connection header received
0928 10:57:01.225199 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:01.225205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:01.225303 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:01.225452 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:01.226097 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:01.226968 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:01.228831 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:01.229741 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:01.229882 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:01.232110 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:01.232115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:01.232119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:01.232141 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:01.232216 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:01.232235 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:01.232403 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:01.232663 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 10:57:01.232733 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:57:01.238309 27978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2267a4b7199487cacd65d135ff1fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2267a4b7199487cacd65d135ff1fe99" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39987 } } }
I20260928 10:57:01.238855 27978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:01.239516 27979 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2267a4b7199487cacd65d135ff1fe99. Latest consensus state: current_term: 1 leader_uuid: "a2267a4b7199487cacd65d135ff1fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2267a4b7199487cacd65d135ff1fe99" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39987 } } }
I20260928 10:57:01.239943 27979 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:01.242570 27984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:01.251297 27984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:01.263900 27984 catalog_manager.cc:1347] Generated new cluster ID: 42e3ae06cdda41ab8450d8e887c9f318
I20260928 10:57:01.264030 27984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:01.280555 27984 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:01.282122 27984 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:01.303223 27984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2267a4b7199487cacd65d135ff1fe99: Generated new TSK 0
I20260928 10:57:01.304265 27984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:57:01.321719 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27900
W20260928 10:57:01.323576 27962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33023
W20260928 10:57:01.324975 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 27900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:57:01.325212 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 27900
W20260928 10:57:01.343897 27885 connection.cc:535] client connection to 127.5.227.190:39987 recv error: Network error: failed to read from TLS socket (remote: 127.5.227.190:39987): Connection reset by peer (error 104)
2026-09-28T10:57:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:01 dist-test-slave-8w8g krb5kdc[27998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:01 dist-test-slave-8w8g krb5kdc[27998](info): set up 2 sockets
Sep 28 10:57:01 dist-test-slave-8w8g krb5kdc[27998](info): commencing operation
krb5kdc: starting...
W20260928 10:57:02.589285 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:57:02 dist-test-slave-8w8g krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:57:02.914669 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37623
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:37623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:03.296976 28011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:03.297377 28011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:03.297562 28011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:03.305339 28011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:03.305553 28011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:03.305673 28011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:03.305786 28011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:03.305876 28011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:03.316486 28011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:37623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:37623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:03.318001 28011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:03.319499 28011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:03.335574 28017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:03.335695 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:03.335749 28019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:03.337266 28011 server_base.cc:1034] running on GCE node
I20260928 10:57:03.338261 28011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:03.340131 28011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:03.341445 28011 hybrid_clock.cc:648] HybridClock initialized: now 1790593023341382 us; error 45 us; skew 500 ppm
I20260928 10:57:03.341864 28011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:03 dist-test-slave-8w8g krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593023, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:03.345553 28011 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:57:03.347283 28011 webserver.cc:460] Webserver started at http://127.5.227.190:35281/ using document root <none> and password file <none>
I20260928 10:57:03.348145 28011 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:03.348275 28011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:03.348552 28011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:03.351832 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "f2db84fafd6e43f2832ad6b497f6947e"
format_stamp: "Formatted at 2026-09-28 10:57:03 on dist-test-slave-8w8g"
server_key: "4966523289af9c83a05d9191a3f54a35"
server_key_iv: "44891dd994b6d284f1b0b79b4652e363"
server_key_version: "encryptionkey@0"
I20260928 10:57:03.352691 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "f2db84fafd6e43f2832ad6b497f6947e"
format_stamp: "Formatted at 2026-09-28 10:57:03 on dist-test-slave-8w8g"
server_key: "4966523289af9c83a05d9191a3f54a35"
server_key_iv: "44891dd994b6d284f1b0b79b4652e363"
server_key_version: "encryptionkey@0"
I20260928 10:57:03.358856 28011 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260928 10:57:03.363394 28026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:03.365351 28011 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:57:03.365553 28011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "f2db84fafd6e43f2832ad6b497f6947e"
format_stamp: "Formatted at 2026-09-28 10:57:03 on dist-test-slave-8w8g"
server_key: "4966523289af9c83a05d9191a3f54a35"
server_key_iv: "44891dd994b6d284f1b0b79b4652e363"
server_key_version: "encryptionkey@0"
I20260928 10:57:03.365765 28011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:03.372767 28011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:03.377641 28011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:03.377928 28011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:03.400112 28011 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:37623
I20260928 10:57:03.400112 28077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:37623 every 8 connection(s)
I20260928 10:57:03.401684 28011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:03.405949 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28011
I20260928 10:57:03.406188 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:57:03.406576 28078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:03.406826 6030 external_mini_cluster.cc:1428] Setting key 634c7818a385b6a98a77bbbb89df601f
I20260928 10:57:03.416287 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:03.408853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42863 (local address 127.5.227.190:37623)
0928 10:57:03.409757 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:03.409769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:03.409817 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:57:03.412114 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:03.412143 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:03.412699 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:03.413110 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:03.414370 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:03.414377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:03.414391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:03.414416 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:03.414507 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:03.414546 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:03.414753 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:03.414869 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 10:57:03.415127 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":425,"threads_started":1}
I20260928 10:57:03.420789 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:57:03.422257 28078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e: Bootstrap starting.
I20260928 10:57:03.426909 28078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:03.428367 28078 log.cc:824] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:03.429652 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:03.426568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38149 (local address 127.5.227.190:37623)
0928 10:57:03.426807 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:03.426815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:03.427390 (+ 575us) server_negotiation.cc:407] Connection header received
0928 10:57:03.427561 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:03.427566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:03.427633 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:03.427798 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:03.428830 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:03.428836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:03.428843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:03.428868 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:03.428928 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:03.428960 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:03.429131 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:03.429291 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 10:57:03.429346 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:57:03.432086 28078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e: No bootstrap required, opened a new log
I20260928 10:57:03.441556 28078 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2db84fafd6e43f2832ad6b497f6947e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37623 } }
I20260928 10:57:03.442046 28078 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:03.442130 28078 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2db84fafd6e43f2832ad6b497f6947e, State: Initialized, Role: FOLLOWER
I20260928 10:57:03.442833 28078 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2db84fafd6e43f2832ad6b497f6947e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37623 } }
I20260928 10:57:03.443094 28078 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:03.443307 28078 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:03.443521 28078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:03.446303 28078 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2db84fafd6e43f2832ad6b497f6947e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37623 } }
I20260928 10:57:03.446959 28078 leader_election.cc:302] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2db84fafd6e43f2832ad6b497f6947e; no voters:
I20260928 10:57:03.447660 28078 leader_election.cc:288] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:03.447893 28088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:03.449509 28088 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [term 1 LEADER]: Becoming Leader. State: Replica: f2db84fafd6e43f2832ad6b497f6947e, State: Running, Role: LEADER
I20260928 10:57:03.450210 28088 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2db84fafd6e43f2832ad6b497f6947e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37623 } }
I20260928 10:57:03.450680 28078 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:03.453677 28089 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2db84fafd6e43f2832ad6b497f6947e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2db84fafd6e43f2832ad6b497f6947e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37623 } } }
I20260928 10:57:03.454183 28089 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:03.453788 28090 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2db84fafd6e43f2832ad6b497f6947e. Latest consensus state: current_term: 1 leader_uuid: "f2db84fafd6e43f2832ad6b497f6947e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2db84fafd6e43f2832ad6b497f6947e" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 37623 } } }
I20260928 10:57:03.455238 28090 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:03.457581 28093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:03.466380 28093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:03.480518 28093 catalog_manager.cc:1347] Generated new cluster ID: 8812e3f9cd9443f09f7bde7271986a7e
I20260928 10:57:03.480665 28093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:03.504427 28093 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:03.505038 28093 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:03.513836 28093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2db84fafd6e43f2832ad6b497f6947e: Generated new TSK 0
I20260928 10:57:03.514966 28093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:57:03.528752 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28011
W20260928 10:57:03.530567 28073 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42863
W20260928 10:57:03.531666 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28011: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:57:03.531900 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28011
2026-09-28T10:57:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:03 dist-test-slave-8w8g krb5kdc[28110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:03 dist-test-slave-8w8g krb5kdc[28110](info): set up 2 sockets
Sep 28 10:57:03 dist-test-slave-8w8g krb5kdc[28110](info): commencing operation
krb5kdc: starting...
W20260928 10:57:04.785341 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:57:05 dist-test-slave-8w8g krb5kdc[28110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:57:05.139880 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44453
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:44453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:05.582404 28123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:05.582931 28123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:05.583174 28123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:05.593005 28123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:05.593160 28123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:05.593247 28123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:05.593341 28123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:05.593413 28123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:05.605651 28123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:44453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:05.607227 28123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:05.609150 28123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:05.625880 28128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:05.625859 28129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:05.626564 28131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:05.627715 28123 server_base.cc:1034] running on GCE node
I20260928 10:57:05.628670 28123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:05.630743 28123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:05.632108 28123 hybrid_clock.cc:648] HybridClock initialized: now 1790593025632148 us; error 131 us; skew 500 ppm
I20260928 10:57:05.632822 28123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:05 dist-test-slave-8w8g krb5kdc[28110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:05.637569 28123 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:57:05.639863 28123 webserver.cc:460] Webserver started at http://127.5.227.190:41863/ using document root <none> and password file <none>
I20260928 10:57:05.640709 28123 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:05.640882 28123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:05.641237 28123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:05.644850 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "6f8cfa80b36343f8919773de1b4f40ca"
format_stamp: "Formatted at 2026-09-28 10:57:05 on dist-test-slave-8w8g"
server_key: "ea390d669a49b476252fce0bb86b724e"
server_key_iv: "b946c3fc2f88df62f6c306d9da85aefc"
server_key_version: "encryptionkey@0"
I20260928 10:57:05.645818 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "6f8cfa80b36343f8919773de1b4f40ca"
format_stamp: "Formatted at 2026-09-28 10:57:05 on dist-test-slave-8w8g"
server_key: "ea390d669a49b476252fce0bb86b724e"
server_key_iv: "b946c3fc2f88df62f6c306d9da85aefc"
server_key_version: "encryptionkey@0"
I20260928 10:57:05.652267 28123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:57:05.657797 28138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:05.660405 28123 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260928 10:57:05.660715 28123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "6f8cfa80b36343f8919773de1b4f40ca"
format_stamp: "Formatted at 2026-09-28 10:57:05 on dist-test-slave-8w8g"
server_key: "ea390d669a49b476252fce0bb86b724e"
server_key_iv: "b946c3fc2f88df62f6c306d9da85aefc"
server_key_version: "encryptionkey@0"
I20260928 10:57:05.660986 28123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:05.680667 28123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:05.686833 28123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:05.687208 28123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:05.715562 28123 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:44453
I20260928 10:57:05.715564 28189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:44453 every 8 connection(s)
I20260928 10:57:05.718312 28123 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:05.724189 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:05.725132 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28123
I20260928 10:57:05.725344 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:57:05.726006 6030 external_mini_cluster.cc:1428] Setting key c013274cb0639e5c0f05e42192415864
I20260928 10:57:05.740656 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca: Bootstrap starting.
I20260928 10:57:05.744439 28190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:05.745636 28190 log.cc:824] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:57:05 dist-test-slave-8w8g krb5kdc[28110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:57:05.749471 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca: No bootstrap required, opened a new log
I20260928 10:57:05.761883 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:05.728357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53595 (local address 127.5.227.190:44453)
0928 10:57:05.729563 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:05.729577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:05.729624 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:57:05.732678 (+ 3054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:05.732716 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:05.733314 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:05.733812 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:05.737324 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:05.740135 (+ 2811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:05.741749 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:05.742529 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:05.742773 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:05.748527 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:05.748537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:05.748554 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:05.748605 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:05.755244 (+ 6639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:05.757103 (+ 1859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:05.757112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:05.757121 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:05.757384 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:05.758036 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:05.758041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:05.758043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:05.758961 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:05.759219 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:05.760021 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 10:57:05.760590 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":495,"threads_started":1}
I20260928 10:57:05.763652 28190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8cfa80b36343f8919773de1b4f40ca" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44453 } }
I20260928 10:57:05.764281 28190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:05.764396 28190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8cfa80b36343f8919773de1b4f40ca, State: Initialized, Role: FOLLOWER
I20260928 10:57:05.765129 28190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8cfa80b36343f8919773de1b4f40ca" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44453 } }
I20260928 10:57:05.765504 28190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:05.765659 28190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:05.765868 28190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:05.768656 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:57:05.771678 28190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8cfa80b36343f8919773de1b4f40ca" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44453 } }
I20260928 10:57:05.772943 28190 leader_election.cc:302] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f8cfa80b36343f8919773de1b4f40ca; no voters:
I20260928 10:57:05.774272 28190 leader_election.cc:288] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:05.774493 28199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:05.777561 28199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [term 1 LEADER]: Becoming Leader. State: Replica: 6f8cfa80b36343f8919773de1b4f40ca, State: Running, Role: LEADER
I20260928 10:57:05.778221 28199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8cfa80b36343f8919773de1b4f40ca" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44453 } }
I20260928 10:57:05.779212 28190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:05.784600 28200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f8cfa80b36343f8919773de1b4f40ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8cfa80b36343f8919773de1b4f40ca" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44453 } } }
I20260928 10:57:05.785141 28200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:05.784580 28201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f8cfa80b36343f8919773de1b4f40ca. Latest consensus state: current_term: 1 leader_uuid: "6f8cfa80b36343f8919773de1b4f40ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8cfa80b36343f8919773de1b4f40ca" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 44453 } } }
I20260928 10:57:05.785288 28201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:05.797688 28209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:05.805131 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:05.775065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49471 (local address 127.5.227.190:44453)
0928 10:57:05.775431 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:05.775479 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:05.776272 (+ 793us) server_negotiation.cc:407] Connection header received
0928 10:57:05.776599 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:05.776614 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:05.776733 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:05.776946 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:05.779682 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:05.781026 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:05.783205 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:05.784033 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:05.784250 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:05.790092 (+ 5842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:05.790104 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:05.790113 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:05.790200 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:05.797046 (+ 6846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:05.799356 (+ 2310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:05.799366 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:05.799373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:05.799657 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:05.800701 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:05.800710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:05.800716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:05.801430 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:05.803799 (+ 2369us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:05.804076 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 10:57:05.804512 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260928 10:57:05.805893 28209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:05.819698 28209 catalog_manager.cc:1347] Generated new cluster ID: bb8e719411f14aa4a77cf431d62d7b90
I20260928 10:57:05.819907 28209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:05.844218 28209 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:05.846297 28209 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:05.863695 28209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f8cfa80b36343f8919773de1b4f40ca: Generated new TSK 0
I20260928 10:57:05.865195 28209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:57:05.879117 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28123
I20260928 10:57:05.881037 28185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:05.988420 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28123
2026-09-28T10:57:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:06 dist-test-slave-8w8g krb5kdc[28222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:06 dist-test-slave-8w8g krb5kdc[28222](info): set up 2 sockets
Sep 28 10:57:06 dist-test-slave-8w8g krb5kdc[28222](info): commencing operation
krb5kdc: starting...
W20260928 10:57:07.224740 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.004s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:57:07 dist-test-slave-8w8g krb5kdc[28222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:57:07.548408 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:34755
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:07.938437 28235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:07.938812 28235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:07.938990 28235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:07.947499 28235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:07.947669 28235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:07.947808 28235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:07.947901 28235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:07.947999 28235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:07.962988 28235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:07.964656 28235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:07.966331 28235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:07.982921 28241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:07.982975 28240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:07.983004 28243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:07.984617 28235 server_base.cc:1034] running on GCE node
I20260928 10:57:07.985682 28235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:07.987470 28235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:07.988817 28235 hybrid_clock.cc:648] HybridClock initialized: now 1790593027988761 us; error 51 us; skew 500 ppm
I20260928 10:57:07.989312 28235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:07 dist-test-slave-8w8g krb5kdc[28222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593027, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:07.993359 28235 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:57:07.995278 28235 webserver.cc:460] Webserver started at http://127.5.227.190:37891/ using document root <none> and password file <none>
I20260928 10:57:07.996177 28235 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:07.996337 28235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:07.996698 28235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:08.000218 28235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "269c88d0c43f40c387abd850fe77b596"
format_stamp: "Formatted at 2026-09-28 10:57:07 on dist-test-slave-8w8g"
server_key: "f45397f3e52b210343235d920a75a12e"
server_key_iv: "33dc59e4068e2d2cd0c5a225c20af2fb"
server_key_version: "encryptionkey@0"
I20260928 10:57:08.001420 28235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "269c88d0c43f40c387abd850fe77b596"
format_stamp: "Formatted at 2026-09-28 10:57:07 on dist-test-slave-8w8g"
server_key: "f45397f3e52b210343235d920a75a12e"
server_key_iv: "33dc59e4068e2d2cd0c5a225c20af2fb"
server_key_version: "encryptionkey@0"
I20260928 10:57:08.008900 28235 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260928 10:57:08.014874 28250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:08.017246 28235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:57:08.017678 28235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "269c88d0c43f40c387abd850fe77b596"
format_stamp: "Formatted at 2026-09-28 10:57:07 on dist-test-slave-8w8g"
server_key: "f45397f3e52b210343235d920a75a12e"
server_key_iv: "33dc59e4068e2d2cd0c5a225c20af2fb"
server_key_version: "encryptionkey@0"
I20260928 10:57:08.017937 28235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:08.056584 28235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:08.062250 28235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:08.062595 28235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:08.084401 28235 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:34755
I20260928 10:57:08.084438 28301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:34755 every 8 connection(s)
I20260928 10:57:08.086035 28235 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:08.090253 28302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:08.091902 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28235
I20260928 10:57:08.092087 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:57:08.092754 6030 external_mini_cluster.cc:1428] Setting key de79bdd9cf010b29690977b8205f8b04
I20260928 10:57:08.105731 28302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596: Bootstrap starting.
I20260928 10:57:08.110275 28302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:08.111215 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:08.095241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46071 (local address 127.5.227.190:34755)
0928 10:57:08.096537 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:08.096552 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:08.096606 (+ 54us) server_negotiation.cc:407] Connection header received
0928 10:57:08.099066 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:08.099096 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:08.099723 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:08.100283 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:08.101049 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:08.102573 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:08.104802 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:08.105977 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:08.106160 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:08.107882 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:08.107896 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:08.107919 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:08.108623 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:08.108715 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:08.108745 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:08.108869 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:08.109421 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 10:57:08.109725 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":740,"threads_started":1}
I20260928 10:57:08.112200 28302 log.cc:824] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:08.115792 28302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596: No bootstrap required, opened a new log
I20260928 10:57:08.116019 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:57:08.125897 28302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269c88d0c43f40c387abd850fe77b596" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34755 } }
I20260928 10:57:08.126523 28302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:08.126660 28302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c88d0c43f40c387abd850fe77b596, State: Initialized, Role: FOLLOWER
I20260928 10:57:08.127383 28302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269c88d0c43f40c387abd850fe77b596" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34755 } }
I20260928 10:57:08.127650 28302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:08.127856 28302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:08.128055 28302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:08.130286 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:08.121780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50351 (local address 127.5.227.190:34755)
0928 10:57:08.122083 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:08.122088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:08.122773 (+ 685us) server_negotiation.cc:407] Connection header received
0928 10:57:08.123080 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:08.123085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:08.123167 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:08.123269 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:08.124033 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:08.124900 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:08.127021 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:08.127897 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:08.128143 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:08.129453 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:08.129465 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:08.129469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:08.129495 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:08.129542 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:08.129562 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:08.129674 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:08.129855 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 10:57:08.129959 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260928 10:57:08.130861 28302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269c88d0c43f40c387abd850fe77b596" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34755 } }
I20260928 10:57:08.131593 28302 leader_election.cc:302] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 269c88d0c43f40c387abd850fe77b596; no voters:
I20260928 10:57:08.132406 28302 leader_election.cc:288] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:08.132614 28312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:08.133636 28312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [term 1 LEADER]: Becoming Leader. State: Replica: 269c88d0c43f40c387abd850fe77b596, State: Running, Role: LEADER
I20260928 10:57:08.134259 28312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269c88d0c43f40c387abd850fe77b596" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34755 } }
I20260928 10:57:08.134974 28302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:08.138842 28313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "269c88d0c43f40c387abd850fe77b596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269c88d0c43f40c387abd850fe77b596" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34755 } } }
I20260928 10:57:08.139482 28314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 269c88d0c43f40c387abd850fe77b596. Latest consensus state: current_term: 1 leader_uuid: "269c88d0c43f40c387abd850fe77b596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269c88d0c43f40c387abd850fe77b596" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 34755 } } }
I20260928 10:57:08.140379 28313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:08.140547 28314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:08.144745 28320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:08.151357 28320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:08.165154 28320 catalog_manager.cc:1347] Generated new cluster ID: e0826d6a31c1410cac231c053de68bd9
I20260928 10:57:08.165329 28320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:08.185490 28320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:08.187013 28320 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:08.196693 28320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 269c88d0c43f40c387abd850fe77b596: Generated new TSK 0
I20260928 10:57:08.198879 28320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:57:08.207752 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28235
W20260928 10:57:08.209503 28297 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46071
W20260928 10:57:08.211091 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28235: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:57:08.211404 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28235
2026-09-28T10:57:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:08 dist-test-slave-8w8g krb5kdc[28333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:08 dist-test-slave-8w8g krb5kdc[28333](info): set up 2 sockets
Sep 28 10:57:08 dist-test-slave-8w8g krb5kdc[28333](info): commencing operation
krb5kdc: starting...
W20260928 10:57:09.414005 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:57:09 dist-test-slave-8w8g krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:57:09.736675 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39455
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:42397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:10.113862 28346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:10.114269 28346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:10.114445 28346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:10.123322 28346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:10.123576 28346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:10.123708 28346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:10.123867 28346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:10.123978 28346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:10.135890 28346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:42397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:10.137430 28346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:10.138909 28346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:10.154558 28354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:10.154559 28351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:10.154366 28352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:10.156126 28346 server_base.cc:1034] running on GCE node
I20260928 10:57:10.157245 28346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:10.159024 28346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:10.160300 28346 hybrid_clock.cc:648] HybridClock initialized: now 1790593030160177 us; error 122 us; skew 500 ppm
I20260928 10:57:10.160802 28346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:10 dist-test-slave-8w8g krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593030, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:10.165350 28346 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:57:10.167649 28346 webserver.cc:460] Webserver started at http://127.5.227.190:45939/ using document root <none> and password file <none>
I20260928 10:57:10.168570 28346 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:10.168753 28346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:10.169116 28346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:10.172847 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "2bca57b5b69e4057b307b2871e83fb71"
format_stamp: "Formatted at 2026-09-28 10:57:10 on dist-test-slave-8w8g"
server_key: "c9da1efa6f2e69abdb33b66a1f33a4d7"
server_key_iv: "2895e56801247bc6039a2273d2ea0c9f"
server_key_version: "encryptionkey@0"
I20260928 10:57:10.173849 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "2bca57b5b69e4057b307b2871e83fb71"
format_stamp: "Formatted at 2026-09-28 10:57:10 on dist-test-slave-8w8g"
server_key: "c9da1efa6f2e69abdb33b66a1f33a4d7"
server_key_iv: "2895e56801247bc6039a2273d2ea0c9f"
server_key_version: "encryptionkey@0"
I20260928 10:57:10.181753 28346 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 10:57:10.187135 28361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:10.189405 28346 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260928 10:57:10.189687 28346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "2bca57b5b69e4057b307b2871e83fb71"
format_stamp: "Formatted at 2026-09-28 10:57:10 on dist-test-slave-8w8g"
server_key: "c9da1efa6f2e69abdb33b66a1f33a4d7"
server_key_iv: "2895e56801247bc6039a2273d2ea0c9f"
server_key_version: "encryptionkey@0"
I20260928 10:57:10.189973 28346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:10.197876 28346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:10.204571 28346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:10.204947 28346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:10.233193 28346 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:39455
I20260928 10:57:10.233224 28412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:39455 every 8 connection(s)
I20260928 10:57:10.234845 28346 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:10.239668 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28346
I20260928 10:57:10.239938 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:57:10.240255 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:10.240691 6030 external_mini_cluster.cc:1428] Setting key e3f034d045044381f1199c4035198efd
I20260928 10:57:10.252151 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:10.242990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60199 (local address 127.5.227.190:39455)
0928 10:57:10.244034 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:10.244052 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:10.244102 (+ 50us) server_negotiation.cc:407] Connection header received
0928 10:57:10.246076 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:10.246107 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:10.246582 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:10.247032 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:10.248833 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:10.248858 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:10.248882 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:10.248905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:10.249063 (+ 158us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:10.249106 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:10.249379 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:10.249865 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 10:57:10.250109 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":478,"threads_started":1}
I20260928 10:57:10.257421 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:57:10.259063 28413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71: Bootstrap starting.
I20260928 10:57:10.263948 28413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:10.265630 28413 log.cc:824] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:10.267359 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:10.263255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32831 (local address 127.5.227.190:39455)
0928 10:57:10.263478 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:10.263486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:10.264598 (+ 1112us) server_negotiation.cc:407] Connection header received
0928 10:57:10.264769 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:10.264775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:10.264881 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:10.265015 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:10.266298 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:10.266303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:10.266306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:57:10.266324 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:10.266390 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:57:10.266415 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:10.266677 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:10.266759 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 10:57:10.266966 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:57:10.269199 28413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71: No bootstrap required, opened a new log
I20260928 10:57:10.277364 28413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bca57b5b69e4057b307b2871e83fb71" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39455 } }
I20260928 10:57:10.277839 28413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:57:10.277959 28413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bca57b5b69e4057b307b2871e83fb71, State: Initialized, Role: FOLLOWER
I20260928 10:57:10.278592 28413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bca57b5b69e4057b307b2871e83fb71" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39455 } }
I20260928 10:57:10.278868 28413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:57:10.279027 28413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:57:10.279199 28413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:10.281687 28413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bca57b5b69e4057b307b2871e83fb71" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39455 } }
I20260928 10:57:10.282299 28413 leader_election.cc:302] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bca57b5b69e4057b307b2871e83fb71; no voters:
I20260928 10:57:10.282958 28413 leader_election.cc:288] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:57:10.283162 28423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:10.284250 28423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [term 1 LEADER]: Becoming Leader. State: Replica: 2bca57b5b69e4057b307b2871e83fb71, State: Running, Role: LEADER
I20260928 10:57:10.285023 28423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bca57b5b69e4057b307b2871e83fb71" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39455 } }
I20260928 10:57:10.285741 28413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:10.288275 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bca57b5b69e4057b307b2871e83fb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bca57b5b69e4057b307b2871e83fb71" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39455 } } }
I20260928 10:57:10.288784 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:10.288241 28425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bca57b5b69e4057b307b2871e83fb71. Latest consensus state: current_term: 1 leader_uuid: "2bca57b5b69e4057b307b2871e83fb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bca57b5b69e4057b307b2871e83fb71" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39455 } } }
I20260928 10:57:10.290688 28425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:10.294850 28430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:10.300640 28430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:10.312664 28430 catalog_manager.cc:1347] Generated new cluster ID: 92fb4195d10c48a09b54a9efee0b330f
I20260928 10:57:10.312810 28430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:10.351276 28430 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:10.351935 28430 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:10.363687 28430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bca57b5b69e4057b307b2871e83fb71: Generated new TSK 0
I20260928 10:57:10.364970 28430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:57:10.382522 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28346
W20260928 10:57:10.384248 28408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60199
W20260928 10:57:10.385103 6030 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:57:10.385447 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28346
2026-09-28T10:57:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2170 ms)
[----------] 18 tests from AuthTokenIssuingTest (40485 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:10 dist-test-slave-8w8g krb5kdc[28444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:10 dist-test-slave-8w8g krb5kdc[28444](info): set up 2 sockets
Sep 28 10:57:10 dist-test-slave-8w8g krb5kdc[28444](info): commencing operation
krb5kdc: starting...
W20260928 10:57:11.625145 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:57:11 dist-test-slave-8w8g krb5kdc[28444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:57:11.949122 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:43451
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:43451,127.5.227.189:35073,127.5.227.188:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:12.338024 28457 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:57:12.338398 28457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:12.338464 28457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:12.338629 28457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:12.346940 28457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:12.347079 28457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:12.347198 28457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:12.347316 28457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:12.347456 28457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:12.359594 28457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:43451,127.5.227.189:35073,127.5.227.188:42231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:12.361274 28457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:12.362742 28457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:12.376964 28462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:12.376901 28465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:12.376961 28463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:12.378239 28457 server_base.cc:1034] running on GCE node
I20260928 10:57:12.379242 28457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:12.380905 28457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:12.382222 28457 hybrid_clock.cc:648] HybridClock initialized: now 1790593032382137 us; error 37 us; skew 500 ppm
I20260928 10:57:12.382851 28457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:12 dist-test-slave-8w8g krb5kdc[28444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:12.397121 28457 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:57:12.399212 28457 webserver.cc:460] Webserver started at http://127.5.227.190:44069/ using document root <none> and password file <none>
I20260928 10:57:12.400095 28457 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:12.400265 28457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:12.400640 28457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:12.404135 28457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "6a23d1fe21bf45c18310c38e25496b56"
format_stamp: "Formatted at 2026-09-28 10:57:12 on dist-test-slave-8w8g"
server_key: "e642a21dc788f245963fae8ae7036a88"
server_key_iv: "f20647dec9b59ce646c7a27954afed85"
server_key_version: "encryptionkey@0"
I20260928 10:57:12.405066 28457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "6a23d1fe21bf45c18310c38e25496b56"
format_stamp: "Formatted at 2026-09-28 10:57:12 on dist-test-slave-8w8g"
server_key: "e642a21dc788f245963fae8ae7036a88"
server_key_iv: "f20647dec9b59ce646c7a27954afed85"
server_key_version: "encryptionkey@0"
I20260928 10:57:12.411966 28457 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:57:12.416646 28472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:12.418654 28457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260928 10:57:12.418936 28457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "6a23d1fe21bf45c18310c38e25496b56"
format_stamp: "Formatted at 2026-09-28 10:57:12 on dist-test-slave-8w8g"
server_key: "e642a21dc788f245963fae8ae7036a88"
server_key_iv: "f20647dec9b59ce646c7a27954afed85"
server_key_version: "encryptionkey@0"
I20260928 10:57:12.419246 28457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:12.432765 28457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:12.439711 28457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:12.440114 28457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:12.466953 28457 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:43451
I20260928 10:57:12.467010 28523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:43451 every 8 connection(s)
I20260928 10:57:12.468660 28457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:12.474097 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28457
I20260928 10:57:12.474295 28524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:12.474408 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:57:12.475425 6030 external_mini_cluster.cc:1428] Setting key cc688837eda2d86fbc1584a0cd2940a2
I20260928 10:57:12.487084 28524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.227.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.189@KRBTEST.COM" created.
Sep 28 10:57:12 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:57:12.527447 28528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.493590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.5.227.190:43451)
0928 10:57:12.494585 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:12.494596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:12.494662 (+ 66us) server_negotiation.cc:407] Connection header received
0928 10:57:12.496844 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:12.496855 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:12.497249 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:12.497549 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:12.499150 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:12.501207 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:12.503348 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:12.504830 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:12.505089 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:12.515386 (+ 10297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:12.515402 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:12.515425 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:12.515488 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:12.521679 (+ 6191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:12.522653 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:12.522666 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:12.522673 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:12.522924 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:12.523559 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:12.523564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:12.523567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:12.524279 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:12.524521 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:12.525180 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 10:57:12.525577 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":714,"thread_start_us":547,"threads_started":1}
I20260928 10:57:12.527487 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.491567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:43451 (local address 127.0.0.1:45912)
0928 10:57:12.493028 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.493091 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:12.496284 (+ 3193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:12.497612 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:12.497618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:12.498310 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:12.498893 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:12.498916 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:12.501580 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:12.501585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:12.503123 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:12.503146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:12.505153 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:12.505160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:12.505317 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:12.512464 (+ 7147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:12.512503 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:12.515040 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:12.521920 (+ 6880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:12.521934 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:12.521947 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:12.522476 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:12.523076 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:12.523084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:12.523090 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:12.523346 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:12.524448 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:12.524457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:12.524709 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 10:57:12.525055 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 10:57:12.525720 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":164,"thread_start_us":886,"threads_started":1}
I20260928 10:57:12.531706 28524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:12.533599 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.532733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37340)
0928 10:57:12.532991 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.533284 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260928 10:57:12.534133 28473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
I20260928 10:57:12.536589 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.536110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37342)
0928 10:57:12.536297 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.536410 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 10:57:12.536893 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35073: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab.
Entry for principal kudu/127.5.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab.
I20260928 10:57:12.566901 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:35073
--webserver_interface=127.5.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab
--principal=kudu/127.5.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:43451,127.5.227.189:35073,127.5.227.188:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:57:12.586690 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } attempt: 1
I20260928 10:57:12.588079 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.587532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37358)
0928 10:57:12.587790 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.587925 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 10:57:12.589433 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.589000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37364)
0928 10:57:12.589191 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.589273 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 10:57:12.589742 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35073: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
I20260928 10:57:12.658015 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } attempt: 2
I20260928 10:57:12.659834 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.658988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37376)
0928 10:57:12.659506 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.659602 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331}
I20260928 10:57:12.661643 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.661124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37388)
0928 10:57:12.661324 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.661411 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 10:57:12.661935 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35073: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
I20260928 10:57:12.753276 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } attempt: 3
I20260928 10:57:12.754813 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.754121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37396)
0928 10:57:12.754382 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.754486 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 10:57:12.756044 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.755568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37398)
0928 10:57:12.755794 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.755890 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 10:57:12.756240 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35073: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
I20260928 10:57:12.899545 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } attempt: 4
I20260928 10:57:12.900897 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.900283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37402)
0928 10:57:12.900639 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.900736 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260928 10:57:12.901928 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:12.901573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37416)
0928 10:57:12.901718 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:12.901803 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 10:57:12.902107 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35073: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35073: connect: Connection refused (error 111)
W20260928 10:57:12.992249 28531 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:57:12.992739 28531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:12.992887 28531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:12.993032 28531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:13.001978 28531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:13.002241 28531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:13.002370 28531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:13.002470 28531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:13.002576 28531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:13.015254 28531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:43451,127.5.227.189:35073,127.5.227.188:42231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab
--principal=kudu/127.5.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:35073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:13.016916 28531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:13.019111 28531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:13.035619 28537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:13.035662 28539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:13.035620 28536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:13.037976 28531 server_base.cc:1034] running on GCE node
I20260928 10:57:13.039579 28531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:13.041790 28531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:13.043136 28531 hybrid_clock.cc:648] HybridClock initialized: now 1790593033043040 us; error 78 us; skew 500 ppm
I20260928 10:57:13.043680 28531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:13.047551 28531 init.cc:377] Logged in from keytab as kudu/127.5.227.189@KRBTEST.COM (short username kudu)
I20260928 10:57:13.049319 28531 webserver.cc:460] Webserver started at http://127.5.227.189:41109/ using document root <none> and password file <none>
I20260928 10:57:13.050149 28531 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:13.050253 28531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:13.050762 28531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:13.055109 28531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data/instance:
uuid: "ed04b61f9f98462d8bbaa24fb95f883b"
format_stamp: "Formatted at 2026-09-28 10:57:13 on dist-test-slave-8w8g"
server_key: "7cf325d983fc4c1f1dd2ebcfb87966d0"
server_key_iv: "09b60da62e1ac85d8307d6c0690f7787"
server_key_version: "encryptionkey@0"
I20260928 10:57:13.056610 28531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/wal/instance:
uuid: "ed04b61f9f98462d8bbaa24fb95f883b"
format_stamp: "Formatted at 2026-09-28 10:57:13 on dist-test-slave-8w8g"
server_key: "7cf325d983fc4c1f1dd2ebcfb87966d0"
server_key_iv: "09b60da62e1ac85d8307d6c0690f7787"
server_key_version: "encryptionkey@0"
I20260928 10:57:13.065682 28531 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260928 10:57:13.072613 28546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:13.079880 28531 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.003s
I20260928 10:57:13.080399 28531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/wal
uuid: "ed04b61f9f98462d8bbaa24fb95f883b"
format_stamp: "Formatted at 2026-09-28 10:57:13 on dist-test-slave-8w8g"
server_key: "7cf325d983fc4c1f1dd2ebcfb87966d0"
server_key_iv: "09b60da62e1ac85d8307d6c0690f7787"
server_key_version: "encryptionkey@0"
I20260928 10:57:13.080720 28531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:13.120612 28531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:13.127319 28531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:13.127671 28531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:13.155231 28531 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.189:35073
I20260928 10:57:13.155174 28597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.189:35073 every 8 connection(s)
I20260928 10:57:13.158298 28531 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
I20260928 10:57:13.162568 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28531
I20260928 10:57:13.162775 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-1/wal/instance
I20260928 10:57:13.163347 6030 external_mini_cluster.cc:1428] Setting key 56d90ff3a9d6663537f8c1e592534cfa
I20260928 10:57:13.163722 28598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:13.175141 28598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:13.201424 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } attempt: 5
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.188@KRBTEST.COM" created.
I20260928 10:57:13.216161 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.178659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.5.227.190:43451)
0928 10:57:13.179195 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.179200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.182016 (+ 2816us) server_negotiation.cc:407] Connection header received
0928 10:57:13.185747 (+ 3731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.185759 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.186125 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.186287 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.191723 (+ 5436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.193801 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.197615 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.198477 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.198688 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.205326 (+ 6638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.205336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.205343 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.205429 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.209312 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.211042 (+ 1730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.211046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.211049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.211209 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.212282 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.212287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.212291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.213672 (+ 1381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.213946 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.215264 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.215561 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":431,"thread_start_us":309,"threads_started":1}
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:57:13.218694 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.179530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:43451 (local address 127.0.0.1:45922)
0928 10:57:13.181786 (+ 2256us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.181841 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.184942 (+ 3101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.187267 (+ 2325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.187287 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.189738 (+ 2451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.191183 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.191226 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.194106 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.194117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.197275 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.197302 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.198803 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.198809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.199530 (+ 721us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.200766 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.200800 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.204572 (+ 3772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.209673 (+ 5101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.209682 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.209699 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.210755 (+ 1056us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.211604 (+ 849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.211609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.211613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.212069 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.213851 (+ 1782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.213861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.214688 (+ 827us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.215075 (+ 387us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.216128 (+ 1053us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1555,"thread_start_us":585,"threads_started":1}
I20260928 10:57:13.221547 28598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:13.230290 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.202382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37424)
0928 10:57:13.202762 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.202789 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.202944 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.204864 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.204870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.205395 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.205703 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.205719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.208238 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.208247 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.209772 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.209786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.211972 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.211978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.212676 (+ 698us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.214184 (+ 1508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.214210 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.218131 (+ 3921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.226297 (+ 8166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.226309 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.226316 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.226864 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.227618 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.227627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.227633 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.227870 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.228751 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.228757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.229003 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.229181 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.229767 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":135}
I20260928 10:57:13.230854 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.202754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.5.227.189:35073)
0928 10:57:13.204028 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.204038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.204080 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:57:13.204167 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.204176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.204569 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.204968 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.205984 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.208024 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.209924 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.211591 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.211829 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.219070 (+ 7241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.219087 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.219108 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.219186 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.225926 (+ 6740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.227036 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.227046 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.227049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.227427 (+ 378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.228004 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.228009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.228012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.228575 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.228838 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.229570 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.230033 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":523,"threads_started":1}
I20260928 10:57:13.234900 28524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:13.236634 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.236003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55662)
0928 10:57:13.236239 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.236331 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:57:13.238057 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.237619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55670)
0928 10:57:13.237826 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.237929 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 10:57:13.238348 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.245510 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.222454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37434)
0928 10:57:13.223136 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.223164 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.223324 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.227086 (+ 3762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.227094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.227989 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.228282 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.228299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.231046 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.231055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.232756 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.232776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.234052 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.234059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.234268 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.235316 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.235412 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.238584 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.242907 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.242913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.242917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.243308 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.243721 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.243762 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.243767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.243941 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.244504 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.244508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.244651 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.244793 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.245121 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296}
I20260928 10:57:13.245643 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.222520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.5.227.189:35073)
0928 10:57:13.226734 (+ 4214us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.226742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.226769 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:57:13.226827 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.226834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.226922 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.227263 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.229037 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.230887 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.232972 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.233856 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.234008 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.238927 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.238933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.238937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.238986 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.242592 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.243458 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.243463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.243468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.243595 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.244062 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.244068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.244076 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.244364 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.244532 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.244897 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.245254 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":4042,"thread_start_us":3849,"threads_started":1}
I20260928 10:57:13.247076 28598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:13.248575 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.248074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55682)
0928 10:57:13.248257 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.248401 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 10:57:13.248847 28547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.251008 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.250570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55694)
0928 10:57:13.250767 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.250878 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 10:57:13.251523 28598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab.
Entry for principal kudu/127.5.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab.
W20260928 10:57:13.272408 6030 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.227.188: real 0.109s user 0.003s sys 0.018s
I20260928 10:57:13.273417 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:42231
--webserver_interface=127.5.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab
--principal=kudu/127.5.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:43451,127.5.227.189:35073,127.5.227.188:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:57:13.289877 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 1
I20260928 10:57:13.291551 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.290973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55704)
0928 10:57:13.291219 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.291336 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 10:57:13.293901 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.293193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55712)
0928 10:57:13.293654 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.293750 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260928 10:57:13.294625 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.301034 28598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 1
I20260928 10:57:13.302457 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.301893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55716)
0928 10:57:13.302153 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.302256 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 10:57:13.303793 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.303300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55728)
0928 10:57:13.303522 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.303607 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 10:57:13.304045 28598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.362973 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 2
I20260928 10:57:13.364710 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.363874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55744)
0928 10:57:13.364178 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.364298 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 10:57:13.365922 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.365534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55752)
0928 10:57:13.365702 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.365792 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 10:57:13.366189 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.372339 28598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 2
I20260928 10:57:13.374496 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.373506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55768)
0928 10:57:13.373949 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.374056 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 10:57:13.376103 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.375634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55770)
0928 10:57:13.375859 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.375959 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 10:57:13.376353 28598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.467857 28598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 3
I20260928 10:57:13.469448 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.468768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55786)
0928 10:57:13.469058 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.469205 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 10:57:13.471292 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.470705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55790)
0928 10:57:13.471013 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.471125 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260928 10:57:13.471642 28598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.472611 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 3
I20260928 10:57:13.474033 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.473400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55798)
0928 10:57:13.473668 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.473782 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 10:57:13.475634 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.475187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55800)
0928 10:57:13.475376 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.475482 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 10:57:13.476040 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.614941 28598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 4
I20260928 10:57:13.616505 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.615840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55816)
0928 10:57:13.616095 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.616236 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 10:57:13.618093 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.617614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55832)
0928 10:57:13.617815 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.617927 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 10:57:13.618589 28598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
I20260928 10:57:13.653317 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 4
I20260928 10:57:13.654862 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.654204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55846)
0928 10:57:13.654471 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.654623 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 10:57:13.656783 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.656203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55862)
0928 10:57:13.656442 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.656570 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 10:57:13.657115 28524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:42231: Network error: Client connection negotiation failed: client connection to 127.5.227.188:42231: connect: Connection refused (error 111)
W20260928 10:57:13.700286 28607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:57:13.700665 28607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:13.700734 28607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:13.700810 28607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:13.710716 28607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:13.710852 28607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:13.710914 28607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:13.710959 28607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:13.711023 28607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:13.722303 28607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:43451,127.5.227.189:35073,127.5.227.188:42231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab
--principal=kudu/127.5.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:13.723809 28607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:13.725535 28607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:13.743078 28613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:13.743173 28612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:13.743192 28615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:13.745785 28607 server_base.cc:1034] running on GCE node
I20260928 10:57:13.746891 28607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:13.748839 28607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:13.750162 28607 hybrid_clock.cc:648] HybridClock initialized: now 1790593033750022 us; error 121 us; skew 500 ppm
I20260928 10:57:13.750659 28607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:13.756122 28607 init.cc:377] Logged in from keytab as kudu/127.5.227.188@KRBTEST.COM (short username kudu)
I20260928 10:57:13.758960 28607 webserver.cc:460] Webserver started at http://127.5.227.188:37943/ using document root <none> and password file <none>
I20260928 10:57:13.760133 28607 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:13.760313 28607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:13.760805 28607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:13.764693 28607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data/instance:
uuid: "26d884a6c9c7456c820e639c0dad129b"
format_stamp: "Formatted at 2026-09-28 10:57:13 on dist-test-slave-8w8g"
server_key: "b116146c0c20505e703720704dd88398"
server_key_iv: "d5e8e852754ec8fe17e403e5f910711f"
server_key_version: "encryptionkey@0"
I20260928 10:57:13.765779 28607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/wal/instance:
uuid: "26d884a6c9c7456c820e639c0dad129b"
format_stamp: "Formatted at 2026-09-28 10:57:13 on dist-test-slave-8w8g"
server_key: "b116146c0c20505e703720704dd88398"
server_key_iv: "d5e8e852754ec8fe17e403e5f910711f"
server_key_version: "encryptionkey@0"
I20260928 10:57:13.776350 28607 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260928 10:57:13.782562 28622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:13.784579 28607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:57:13.785143 28607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/wal
uuid: "26d884a6c9c7456c820e639c0dad129b"
format_stamp: "Formatted at 2026-09-28 10:57:13 on dist-test-slave-8w8g"
server_key: "b116146c0c20505e703720704dd88398"
server_key_iv: "d5e8e852754ec8fe17e403e5f910711f"
server_key_version: "encryptionkey@0"
I20260928 10:57:13.785554 28607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:13.808967 28607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:13.816780 28607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:13.817145 28607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:13.844828 28607 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.188:42231
I20260928 10:57:13.844992 28673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.188:42231 every 8 connection(s)
I20260928 10:57:13.846459 28607 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
I20260928 10:57:13.848861 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28607
I20260928 10:57:13.849105 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592619723274-6030-0/minicluster-data/master-2/wal/instance
I20260928 10:57:13.849601 6030 external_mini_cluster.cc:1428] Setting key 9b3c3e46260a7a745a1d0a5a67f2a9b2
I20260928 10:57:13.852536 28674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:13.863716 28674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:57:13.884555 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.852202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.5.227.188:42231)
0928 10:57:13.854245 (+ 2043us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.854261 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.854330 (+ 69us) server_negotiation.cc:407] Connection header received
0928 10:57:13.856957 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.856994 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.857857 (+ 863us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.858415 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.860919 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.862982 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.865093 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.866519 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.866754 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.873188 (+ 6434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.873202 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.873220 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.873276 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.878252 (+ 4976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.879670 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.879681 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.879696 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.880051 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.880589 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.880595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.880599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.881238 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.881780 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.882387 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.882902 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756,"thread_start_us":866,"threads_started":1}
I20260928 10:57:13.888458 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.867695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:43451 (local address 127.0.0.1:45924)
0928 10:57:13.869032 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.869100 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.869332 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.870216 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.870222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.871103 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.871418 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.871438 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.873071 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.873081 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.874483 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.874515 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.875618 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.875631 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.875871 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.876812 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.876846 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.880456 (+ 3610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.884449 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.884461 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.884469 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.885414 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.885916 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.885922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.885926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.886073 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.886839 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.886860 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.887304 (+ 444us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.887551 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.888043 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":526,"threads_started":1}
I20260928 10:57:13.888520 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.867809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.5.227.190:43451)
0928 10:57:13.869738 (+ 1929us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.869746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.869771 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:57:13.869853 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.869861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.869960 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.870116 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.871620 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.872873 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.874712 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.875135 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.875304 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.880766 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.880772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.880775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.880816 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.884264 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.885572 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.885579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.885584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.885746 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.886225 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.886229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.886231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.886655 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.886902 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.887794 (+ 892us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.888096 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1772,"thread_start_us":1525,"threads_started":1}
I20260928 10:57:13.891031 28674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:57:13.910679 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.892231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35073 (local address 127.0.0.1:37440)
0928 10:57:13.892506 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.892527 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.892686 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.893667 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.893672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.894535 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.894880 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.894895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.896951 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.896956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.898356 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.898370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.899926 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.899934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.900073 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.900984 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.901005 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.904224 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.907973 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.907979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.907989 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.908415 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.908856 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.908861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.908865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.909030 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.909629 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.909634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.909817 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.909977 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.910298 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 10:57:13.910713 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.892397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.5.227.189:35073)
0928 10:57:13.893145 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.893150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.893167 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:57:13.893219 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.893223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.893282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.893392 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.895136 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.896736 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.898659 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.899542 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.899708 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.904520 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.904533 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.904536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.904582 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.907698 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.908564 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.908570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.908573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.908727 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.909152 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.909158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.909161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.909497 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.909676 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.910079 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.910323 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":355,"threads_started":1}
I20260928 10:57:13.912276 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.889028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.5.227.190:43451)
0928 10:57:13.889702 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.889707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.889721 (+ 14us) server_negotiation.cc:407] Connection header received
0928 10:57:13.889768 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.889771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.889880 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.890171 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.893109 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.894831 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.897259 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.897972 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.898126 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.904623 (+ 6497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.904629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.904633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.904678 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.908208 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.909111 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.909116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.909118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.909271 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.909944 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.909949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.909951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.910307 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.910743 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.911298 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.911707 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567}
I20260928 10:57:13.912551 28674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:13.917867 28598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 5
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
I20260928 10:57:13.934182 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.913866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55870)
0928 10:57:13.914294 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.914316 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.914566 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.915248 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.915253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.916125 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.916586 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.916606 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.918939 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.918945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.920953 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.920972 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.921971 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.921978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.922227 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.923332 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.923364 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.926411 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.930322 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.930330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.930334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.931018 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.931715 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.931721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.931724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.932085 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.932964 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.932970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.933163 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.933378 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.933730 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 10:57:13.934356 28524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } attempt: 5
I20260928 10:57:13.937887 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.914311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.5.227.188:42231)
0928 10:57:13.914613 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.914617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.914707 (+ 90us) server_negotiation.cc:407] Connection header received
0928 10:57:13.914823 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.914827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.914932 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.915058 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.916843 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.918765 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.921135 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.921766 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.921893 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.926625 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.926631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.926635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.926683 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.930137 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.931182 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.931189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.931194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.931540 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.932224 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.932228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.932234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.932662 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.932894 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.936768 (+ 3874us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.937275 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:57:13.942253 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.915257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.5.227.189:35073)
0928 10:57:13.915597 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.915602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.915850 (+ 248us) server_negotiation.cc:407] Connection header received
0928 10:57:13.916008 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.916013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.916085 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.916176 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.918530 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.919933 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.921796 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.922677 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.922807 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.930045 (+ 7238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.930053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.930059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.930102 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.934945 (+ 4843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.937054 (+ 2109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.937063 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.937069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.937256 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.938289 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.938294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.938297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.939156 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.939781 (+ 625us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.940804 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.941889 (+ 1085us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260928 10:57:13.944729 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.918733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55872)
0928 10:57:13.919082 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.919107 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.919461 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.920615 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.920625 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.921209 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.921618 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.921632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.924821 (+ 3189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.924831 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.926830 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.926854 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.927900 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.927906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.928083 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.928816 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.928989 (+ 173us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.933793 (+ 4804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.940950 (+ 7157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.940956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.940959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.941683 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.942204 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.942210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.942216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.942441 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.943309 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.943314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.943496 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.943891 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.944263 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260928 10:57:13.944833 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.918790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.5.227.188:42231)
0928 10:57:13.920217 (+ 1427us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.920228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.920260 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:57:13.920316 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.920321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.920410 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.920533 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.923024 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.924645 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.927023 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.927625 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.927798 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.934091 (+ 6293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.934096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.934099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.934133 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.940780 (+ 6647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.941837 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.941843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.941846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.942027 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.942643 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.942648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.942657 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.943098 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.943491 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.943931 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.944183 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":1009,"threads_started":1}
I20260928 10:57:13.947317 28674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: Bootstrap starting.
I20260928 10:57:13.951236 28598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: Bootstrap starting.
I20260928 10:57:13.952872 28674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:13.955204 28674 log.cc:824] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:57:13 dist-test-slave-8w8g krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
I20260928 10:57:13.958185 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:57:13.960225 28598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:13.963094 28598 log.cc:824] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:13.965327 28674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: No bootstrap required, opened a new log
I20260928 10:57:13.962877 28499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6a23d1fe21bf45c18310c38e25496b56"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45938
I20260928 10:57:13.969055 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.935027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:42231 (local address 127.0.0.1:55874)
0928 10:57:13.935264 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:13.935286 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:13.935529 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:13.937881 (+ 2352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:13.937888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:13.938940 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:13.939336 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.939360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.950193 (+ 10833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.950201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.951909 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:13.951934 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.952958 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.952966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:13.953194 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.953931 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:13.953953 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:13.958516 (+ 4563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:13.963711 (+ 5195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.963721 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.963725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.964843 (+ 1118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.965638 (+ 795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:13.965652 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:13.965658 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:13.965952 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:13.967024 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:13.967033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:13.967287 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 10:57:13.967635 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 10:57:13.968257 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 10:57:13.970160 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:13.935659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.5.227.188:42231)
0928 10:57:13.937235 (+ 1576us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:13.937243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:13.937269 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:57:13.937334 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:13.937510 (+ 176us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:13.937627 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:13.948088 (+ 10461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:13.948429 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.949977 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.952257 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:13.952759 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:13.952927 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:13.958873 (+ 5946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:13.958880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:13.958883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:13.958921 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:13.963244 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.965137 (+ 1893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.965146 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.965153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.965438 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:13.966147 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:13.966156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:13.966162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:13.966726 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:13.966916 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:13.968917 (+ 2001us) server_negotiation.cc:299] Negotiation successful
0928 10:57:13.969318 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":795,"threads_started":1}
I20260928 10:57:13.974046 28598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: No bootstrap required, opened a new log
I20260928 10:57:13.978425 28524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56: Bootstrap starting.
I20260928 10:57:13.980163 28674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
I20260928 10:57:13.981019 28674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d884a6c9c7456c820e639c0dad129b, State: Initialized, Role: FOLLOWER
I20260928 10:57:13.981855 28674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
I20260928 10:57:13.985231 28524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:13.987404 28524 log.cc:824] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:13.989117 28598 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
I20260928 10:57:13.989630 28598 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed04b61f9f98462d8bbaa24fb95f883b, State: Initialized, Role: FOLLOWER
I20260928 10:57:13.989430 28685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:13.989897 28685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:13.990484 28598 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
I20260928 10:57:13.991080 28674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:13.991916 28687 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:13.992425 28687 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:13.993249 28598 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:13.995086 28524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56: No bootstrap required, opened a new log
I20260928 10:57:14.010354 28524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
I20260928 10:57:14.011267 28524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a23d1fe21bf45c18310c38e25496b56, State: Initialized, Role: FOLLOWER
I20260928 10:57:14.012514 28524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
W20260928 10:57:14.012979 28701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:57:14.013545 28701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:57:14.008023 28531 master_runner.cc:186] Error getting master registration for 127.5.227.190:43451: OK, instance_id { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" instance_seqno: 1790593032458537 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:57:14.026955 28531 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:57:14.028285 28703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.028777 28703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:14.029984 28524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:14.030108 28499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 10:57:14.030330 28499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:14.035878 28499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
I20260928 10:57:14.039506 28573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a23d1fe21bf45c18310c38e25496b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed04b61f9f98462d8bbaa24fb95f883b"
I20260928 10:57:14.039803 28499 leader_election.cc:288] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [CANDIDATE]: Term 1 election: Requested vote from peers ed04b61f9f98462d8bbaa24fb95f883b (127.5.227.189:35073), 26d884a6c9c7456c820e639c0dad129b (127.5.227.188:42231)
I20260928 10:57:14.040159 28573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:14.043613 28573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a23d1fe21bf45c18310c38e25496b56 in term 1.
I20260928 10:57:14.044835 28473 leader_election.cc:302] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a23d1fe21bf45c18310c38e25496b56, ed04b61f9f98462d8bbaa24fb95f883b; no voters:
I20260928 10:57:14.045069 28649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a23d1fe21bf45c18310c38e25496b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "26d884a6c9c7456c820e639c0dad129b"
I20260928 10:57:14.045682 28703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:14.045769 28649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:14.056143 28703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [term 1 LEADER]: Becoming Leader. State: Replica: 6a23d1fe21bf45c18310c38e25496b56, State: Running, Role: LEADER
I20260928 10:57:14.057318 28703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } }
I20260928 10:57:14.055671 28649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a23d1fe21bf45c18310c38e25496b56 in term 1.
W20260928 10:57:14.069867 28628 tablet.cc:2369] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:14.070777 28710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a23d1fe21bf45c18310c38e25496b56. Latest consensus state: current_term: 1 leader_uuid: "6a23d1fe21bf45c18310c38e25496b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.072654 28710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:14.089038 28720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:14.123183 28720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 10:57:14.130999 28552 tablet.cc:2369] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:14.154047 28729 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:57:14.166421 28649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [term 1 FOLLOWER]: Refusing update from remote peer 6a23d1fe21bf45c18310c38e25496b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:14.168813 28573 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [term 1 FOLLOWER]: Refusing update from remote peer 6a23d1fe21bf45c18310c38e25496b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:14.170516 28703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 10:57:14.171700 28703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:14.185982 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:14.092975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.5.227.189:35073)
0928 10:57:14.093229 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.093238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.100591 (+ 7353us) server_negotiation.cc:407] Connection header received
0928 10:57:14.103097 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.103107 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.103233 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.103334 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.109434 (+ 6100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:14.111033 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:14.121805 (+ 10772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:14.123178 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:14.123368 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:14.145156 (+ 21788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:14.145168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:14.145176 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:14.145249 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:14.151830 (+ 6581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:14.173556 (+ 21726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:14.173567 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:14.173574 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:14.173889 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:14.179935 (+ 6046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:14.179946 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:14.179953 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:14.180591 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:14.181100 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:14.184250 (+ 3150us) server_negotiation.cc:299] Negotiation successful
0928 10:57:14.185270 (+ 1020us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:57:14.191577 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:14.075667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.5.227.190:43451)
0928 10:57:14.075980 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.075990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.092908 (+ 16918us) server_negotiation.cc:407] Connection header received
0928 10:57:14.093484 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.093497 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.093619 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.093745 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.111929 (+ 18184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:14.114887 (+ 2958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:14.125154 (+ 10267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:14.126361 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:14.126692 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:14.153887 (+ 27195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:14.153899 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:14.153908 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:14.153978 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:14.171842 (+ 17864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:14.182182 (+ 10340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:14.182194 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:14.182200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:14.182639 (+ 439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:14.188005 (+ 5366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:14.188014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:14.188019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:14.188574 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:14.188991 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:14.190544 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0928 10:57:14.191131 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260928 10:57:14.193768 28478 tablet.cc:2369] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:14.202852 28736 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:57:14.209877 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:14.101758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.5.227.188:42231)
0928 10:57:14.102017 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.102026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.112255 (+ 10229us) server_negotiation.cc:407] Connection header received
0928 10:57:14.115425 (+ 3170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.115435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.115601 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.115700 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.131939 (+ 16239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:14.133557 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:14.141061 (+ 7504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:14.142801 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:14.143031 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:14.198328 (+ 55297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:14.198336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:14.198342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:14.198413 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:14.204160 (+ 5747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:14.206196 (+ 2036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:14.206203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:14.206207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:14.206417 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:14.207252 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:14.207258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:14.207261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:14.207982 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:14.208296 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:14.208909 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 10:57:14.209174 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 10:57:14.163121 28607 master_runner.cc:186] Error getting master registration for 127.5.227.190:43451: OK, instance_id { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" instance_seqno: 1790593032458537 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:57:14.216320 28687 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a23d1fe21bf45c18310c38e25496b56. Latest consensus state: current_term: 1 leader_uuid: "6a23d1fe21bf45c18310c38e25496b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.216646 28607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:57:14.216720 28687 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:14.221246 28685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a23d1fe21bf45c18310c38e25496b56. Latest consensus state: current_term: 1 leader_uuid: "6a23d1fe21bf45c18310c38e25496b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.221768 28685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:14.227650 28730 mvcc.cc:205] Tried to move back new op lower bound from 7334269067929858048 to 7334269067514277888. Current Snapshot: MvccSnapshot[applied={T|T < 7334269067929858048}]
I20260928 10:57:14.233775 28710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a23d1fe21bf45c18310c38e25496b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.234130 28710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:14.238091 28720 catalog_manager.cc:1347] Generated new cluster ID: 3b90196c30c94d6e8644e01f010790b8
I20260928 10:57:14.238224 28720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:14.239148 28703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a23d1fe21bf45c18310c38e25496b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.239494 28703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:14.241726 28733 mvcc.cc:205] Tried to move back new op lower bound from 7334269067929858048 to 7334269067514277888. Current Snapshot: MvccSnapshot[applied={T|T < 7334269067929858048}]
I20260928 10:57:14.242937 28685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a23d1fe21bf45c18310c38e25496b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.243237 28685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:14.258883 28687 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a23d1fe21bf45c18310c38e25496b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a23d1fe21bf45c18310c38e25496b56" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 43451 } } peers { permanent_uuid: "ed04b61f9f98462d8bbaa24fb95f883b" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35073 } } peers { permanent_uuid: "26d884a6c9c7456c820e639c0dad129b" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 42231 } } }
I20260928 10:57:14.259161 28687 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:14.286721 28720 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:14.290129 28720 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:14.319110 28720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a23d1fe21bf45c18310c38e25496b56: Generated new TSK 0
I20260928 10:57:14.320575 28720 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:57:14.422396 28679 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.419172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.5.227.190:43451)
0928 10:57:14.420363 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.420380 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.420410 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:57:14.420488 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.420494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.420582 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.420700 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.422110 (+ 1410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45962: BlockingRecv error: recv got EOF from 127.0.0.1:45962 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 10:57:14.424686 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.421576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.5.227.188:42231)
0928 10:57:14.421748 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.421755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.422133 (+ 378us) server_negotiation.cc:407] Connection header received
0928 10:57:14.422500 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.422517 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.422620 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.422713 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.424340 (+ 1627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55884: BlockingRecv error: recv got EOF from 127.0.0.1:55884 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 10:57:14.427888 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.420229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.5.227.189:35073)
0928 10:57:14.420568 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.420577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.420669 (+ 92us) server_negotiation.cc:407] Connection header received
0928 10:57:14.425945 (+ 5276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.425952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.426051 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.426133 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.427568 (+ 1435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37468: BlockingRecv error: recv got EOF from 127.0.0.1:37468 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
W20260928 10:57:14.449985 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.446946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.5.227.189:35073)
0928 10:57:14.447176 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.447182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.447923 (+ 741us) server_negotiation.cc:407] Connection header received
0928 10:57:14.448177 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.448184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.448349 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.448656 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.449660 (+ 1004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37484: BlockingRecv error: recv got EOF from 127.0.0.1:37484 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 10:57:14.461519 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.458608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.5.227.189:35073)
0928 10:57:14.458818 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.458824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.459639 (+ 815us) server_negotiation.cc:407] Connection header received
0928 10:57:14.459779 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.459877 (+ 98us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.460010 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.460155 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.461273 (+ 1118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37494: BlockingRecv error: recv got EOF from 127.0.0.1:37494 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 10:57:14.507483 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.500813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.5.227.189:35073)
0928 10:57:14.501170 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.501177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.505229 (+ 4052us) server_negotiation.cc:407] Connection header received
0928 10:57:14.505778 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.505784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.505895 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.506012 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.507205 (+ 1193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37500: BlockingRecv error: recv got EOF from 127.0.0.1:37500 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 10:57:14.517988 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.515375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.5.227.189:35073)
0928 10:57:14.515649 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.515656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.516296 (+ 640us) server_negotiation.cc:407] Connection header received
0928 10:57:14.516520 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.516526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.516625 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.516745 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.517717 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37514: BlockingRecv error: recv got EOF from 127.0.0.1:37514 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260928 10:57:14.545161 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.542214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.5.227.189:35073)
0928 10:57:14.542425 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.542432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.543115 (+ 683us) server_negotiation.cc:407] Connection header received
0928 10:57:14.543342 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.543358 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.543516 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.543656 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.544874 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37516: BlockingRecv error: recv got EOF from 127.0.0.1:37516 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 10:57:14.569742 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.566296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.5.227.189:35073)
0928 10:57:14.566826 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.566833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.567564 (+ 731us) server_negotiation.cc:407] Connection header received
0928 10:57:14.567873 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.567881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.567999 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.568128 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.569300 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37530: BlockingRecv error: recv got EOF from 127.0.0.1:37530 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 10:57:14.629024 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.625788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.5.227.189:35073)
0928 10:57:14.626110 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.626117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.627142 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 10:57:14.627335 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.627342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.627490 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.627638 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.628701 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37536: BlockingRecv error: recv got EOF from 127.0.0.1:37536 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260928 10:57:14.703198 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.700195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.5.227.189:35073)
0928 10:57:14.700384 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.700391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.700999 (+ 608us) server_negotiation.cc:407] Connection header received
0928 10:57:14.701216 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.701222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.701373 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.701508 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.702636 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37550: BlockingRecv error: recv got EOF from 127.0.0.1:37550 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 10:57:14.840039 28680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:14.836851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.5.227.189:35073)
0928 10:57:14.837230 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:14.837240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:14.838084 (+ 844us) server_negotiation.cc:407] Connection header received
0928 10:57:14.838251 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:14.838255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:14.838339 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:14.838577 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:14.839563 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37554: BlockingRecv error: recv got EOF from 127.0.0.1:37554 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260928 10:57:15.016366 28701 catalog_manager.cc:1259] Loaded cluster ID: 3b90196c30c94d6e8644e01f010790b8
I20260928 10:57:15.016602 28701 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: loading cluster ID for follower catalog manager: success
I20260928 10:57:15.020298 28701 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: acquiring CA information for follower catalog manager: success
I20260928 10:57:15.022512 28701 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed04b61f9f98462d8bbaa24fb95f883b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:57:15.109095 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:15.102236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.5.227.189:35073)
0928 10:57:15.102616 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:15.102623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:15.102871 (+ 248us) server_negotiation.cc:407] Connection header received
0928 10:57:15.103135 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:15.103140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:15.103299 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:15.103433 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:15.103958 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.104690 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.106171 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.106616 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.106790 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:15.107770 (+ 980us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:15.108413 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:15.108628 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:15.108695 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 10:57:15.108801 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
W20260928 10:57:15.121758 28808 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:15.118790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.5.227.188:42231)
0928 10:57:15.120086 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:15.120091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:15.120110 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:57:15.120211 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:15.120215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:15.120323 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:15.120443 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:15.121468 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55886: BlockingRecv error: recv got EOF from 127.0.0.1:55886 (error 108)
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":1066,"threads_started":1}
I20260928 10:57:15.162237 28729 catalog_manager.cc:1259] Loaded cluster ID: 3b90196c30c94d6e8644e01f010790b8
I20260928 10:57:15.162398 28729 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: loading cluster ID for follower catalog manager: success
I20260928 10:57:15.167104 28729 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: acquiring CA information for follower catalog manager: success
I20260928 10:57:15.170002 28729 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 26d884a6c9c7456c820e639c0dad129b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:57:15.644914 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:15.638339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.5.227.189:35073)
0928 10:57:15.638978 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:15.638988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:15.639415 (+ 427us) server_negotiation.cc:407] Connection header received
0928 10:57:15.639819 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:15.639823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:15.639897 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:15.640025 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:15.640713 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.641506 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.643000 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.643433 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.643598 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:15.644087 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:15.644206 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:15.644311 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:15.644489 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 10:57:15.644581 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":382,"threads_started":1}
I20260928 10:57:15.661765 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:15.652810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.5.227.188:42231)
0928 10:57:15.653514 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:15.653520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:15.653703 (+ 183us) server_negotiation.cc:407] Connection header received
0928 10:57:15.653913 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:15.653917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:15.654076 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:15.654205 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:15.655150 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.656574 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.658846 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.659287 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.659540 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:15.660278 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:15.661010 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:15.661149 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:15.661413 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 10:57:15.661500 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":329,"threads_started":1}
I20260928 10:57:15.680258 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:15.671626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.5.227.188:42231)
0928 10:57:15.671915 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:15.671921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:15.672409 (+ 488us) server_negotiation.cc:407] Connection header received
0928 10:57:15.672646 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:15.672650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:15.672717 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:15.672817 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:15.673373 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.674428 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.677244 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.678261 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.678420 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:15.679267 (+ 847us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:15.679424 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:15.679498 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:15.679821 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 10:57:15.679927 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:57:15.680330 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:15.670656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.5.227.189:35073)
0928 10:57:15.670883 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:15.670890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:15.671494 (+ 604us) server_negotiation.cc:407] Connection header received
0928 10:57:15.672165 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:15.672170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:15.672274 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:15.672344 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:15.673780 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.674816 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.677678 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.678575 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.678735 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:15.679320 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:15.679466 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:15.679541 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:15.679889 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 10:57:15.679976 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:57:15.681715 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:15.669781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.5.227.190:43451)
0928 10:57:15.670975 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:15.670980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:15.670997 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:57:15.671102 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:15.671106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:15.671270 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:15.672207 (+ 937us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:15.672649 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.674137 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.677041 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:15.678223 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:15.678421 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:15.679531 (+ 1110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:15.680206 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:15.680554 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:15.681164 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 10:57:15.681269 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":938,"threads_started":1}
I20260928 10:57:15.689157 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28457
I20260928 10:57:15.691152 28519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:15.825747 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28457
I20260928 10:57:15.854756 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28531
I20260928 10:57:15.856184 28593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:15.986553 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28531
I20260928 10:57:16.012459 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28607
I20260928 10:57:16.013780 28669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:16.135653 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28607
2026-09-28T10:57:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5758 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:57:16 dist-test-slave-8w8g krb5kdc[28836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:57:16 dist-test-slave-8w8g krb5kdc[28836](info): set up 2 sockets
Sep 28 10:57:16 dist-test-slave-8w8g krb5kdc[28836](info): commencing operation
krb5kdc: starting...
W20260928 10:57:17.369395 6030 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/test-user.keytab.
Sep 28 10:57:17 dist-test-slave-8w8g krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
Entry for principal kudu/127.5.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab.
I20260928 10:57:17.725888 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39835
--webserver_interface=127.5.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39835,127.5.227.189:35523,127.5.227.188:44877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:57:18.182214 28849 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:57:18.182714 28849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:18.182871 28849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:18.183029 28849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:18.192261 28849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:18.192467 28849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:18.192595 28849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:18.192698 28849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:18.192799 28849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:18.213060 28849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39835,127.5.227.189:35523,127.5.227.188:44877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.190.keytab
--principal=kudu/127.5.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.190:39835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:18.215791 28849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:18.218024 28849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 10:57:18.233989 28849 server_base.cc:1034] running on GCE node
W20260928 10:57:18.233856 28854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:18.234110 28855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:18.234597 28857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:18.236096 28849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:18.239058 28849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:18.240548 28849 hybrid_clock.cc:648] HybridClock initialized: now 1790593038240445 us; error 71 us; skew 500 ppm
I20260928 10:57:18.241029 28849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:18 dist-test-slave-8w8g krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:18.250802 28849 init.cc:377] Logged in from keytab as kudu/127.5.227.190@KRBTEST.COM (short username kudu)
I20260928 10:57:18.254706 28849 webserver.cc:460] Webserver started at http://127.5.227.190:41267/ using document root <none> and password file <none>
I20260928 10:57:18.255903 28849 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:18.256101 28849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:18.256510 28849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:18.261425 28849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data/instance:
uuid: "91c9720cc44c429c96370690be6aa1e9"
format_stamp: "Formatted at 2026-09-28 10:57:18 on dist-test-slave-8w8g"
server_key: "9757bde1c411ac36c1cee01023d9b7fc"
server_key_iv: "b9fd29c3636a4d0289e6270e0185ac44"
server_key_version: "encryptionkey@0"
I20260928 10:57:18.262470 28849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/wal/instance:
uuid: "91c9720cc44c429c96370690be6aa1e9"
format_stamp: "Formatted at 2026-09-28 10:57:18 on dist-test-slave-8w8g"
server_key: "9757bde1c411ac36c1cee01023d9b7fc"
server_key_iv: "b9fd29c3636a4d0289e6270e0185ac44"
server_key_version: "encryptionkey@0"
I20260928 10:57:18.269155 28849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:57:18.274132 28864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:18.276546 28849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:57:18.276850 28849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/wal
uuid: "91c9720cc44c429c96370690be6aa1e9"
format_stamp: "Formatted at 2026-09-28 10:57:18 on dist-test-slave-8w8g"
server_key: "9757bde1c411ac36c1cee01023d9b7fc"
server_key_iv: "b9fd29c3636a4d0289e6270e0185ac44"
server_key_version: "encryptionkey@0"
I20260928 10:57:18.277179 28849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:18.300350 28849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:18.307077 28849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:18.307447 28849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:18.332899 28915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.190:39835 every 8 connection(s)
I20260928 10:57:18.333014 28849 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.190:39835
I20260928 10:57:18.335094 28849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/data/info.pb
I20260928 10:57:18.340549 28916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:18.343881 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28849
I20260928 10:57:18.344117 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-0/wal/instance
I20260928 10:57:18.344725 6030 external_mini_cluster.cc:1428] Setting key bd7d97cbee3b861cebe4ca3a09f39dd6
I20260928 10:57:18.353652 28916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:57:18 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.227.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.189@KRBTEST.COM" created.
I20260928 10:57:18.383369 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.359569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.5.227.190:39835)
0928 10:57:18.360715 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:18.360737 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:18.361230 (+ 493us) server_negotiation.cc:407] Connection header received
0928 10:57:18.363722 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:18.363783 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:18.364140 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:18.364394 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:18.365858 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:18.367231 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:18.368903 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:18.369514 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:18.369744 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:18.374037 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:18.374047 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:18.374061 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:18.374097 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:18.378096 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:18.378959 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:18.378965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:18.378968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:18.379123 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:18.379554 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:18.379559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:18.379562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:18.380277 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:18.380530 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:18.381303 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 10:57:18.382086 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":346,"server-negotiator.queue_time_us":868,"thread_start_us":526,"threads_started":1}
I20260928 10:57:18.384375 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.359635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39835 (local address 127.0.0.1:36072)
0928 10:57:18.361047 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.361089 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:18.363219 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:18.364418 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:18.364425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:18.365225 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:18.365598 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:18.365618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:18.367478 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:18.367484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:18.368720 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:18.368735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:18.369795 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:18.369802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:18.369995 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:18.370773 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:18.370984 (+ 211us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:18.373740 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:18.378303 (+ 4563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:18.378316 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:18.378330 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:18.378781 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:18.379250 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:18.379258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:18.379261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:18.379430 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:18.380445 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:18.380456 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:18.380738 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 10:57:18.381070 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 10:57:18.381793 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"mutex_wait_us":1,"spinlock_wait_cycles":76160,"thread_start_us":987,"threads_started":1}
I20260928 10:57:18.387603 28916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:18.389206 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.388596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45300)
0928 10:57:18.388850 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.389015 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 10:57:18.389673 28867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
I20260928 10:57:18.391690 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.391231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45314)
0928 10:57:18.391424 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.391503 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 10:57:18.392078 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35523: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab.
Entry for principal kudu/127.5.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab.
I20260928 10:57:18.432883 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:35523
--webserver_interface=127.5.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab
--principal=kudu/127.5.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39835,127.5.227.189:35523,127.5.227.188:44877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:57:18.441448 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } attempt: 1
I20260928 10:57:18.442978 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.442279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45326)
0928 10:57:18.442646 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.442836 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 10:57:18.444111 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.443694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45338)
0928 10:57:18.443879 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.443964 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 10:57:18.444380 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35523: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
I20260928 10:57:18.512727 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } attempt: 2
I20260928 10:57:18.514521 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.513641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45350)
0928 10:57:18.514095 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.514215 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 10:57:18.516628 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.516049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45352)
0928 10:57:18.516288 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.516468 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 10:57:18.516947 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35523: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
I20260928 10:57:18.608322 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } attempt: 3
I20260928 10:57:18.609980 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.609454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45358)
0928 10:57:18.609686 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.609783 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 10:57:18.611202 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.610799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45368)
0928 10:57:18.610964 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.611050 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 10:57:18.611469 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35523: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
I20260928 10:57:18.754953 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } attempt: 4
I20260928 10:57:18.756705 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.755967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45374)
0928 10:57:18.756410 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.756545 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 10:57:18.758466 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:18.757877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45376)
0928 10:57:18.758092 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:18.758217 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 10:57:18.759181 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.189:35523: Network error: Client connection negotiation failed: client connection to 127.5.227.189:35523: connect: Connection refused (error 111)
W20260928 10:57:18.862428 28923 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:57:18.862846 28923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:18.862931 28923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:18.863027 28923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:18.872628 28923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:18.872766 28923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:18.872843 28923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:18.872902 28923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:18.872972 28923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:18.886801 28923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39835,127.5.227.189:35523,127.5.227.188:44877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.189.keytab
--principal=kudu/127.5.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.189:35523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:18.888422 28923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:18.889887 28923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:18.905781 28928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:18.905781 28929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:18.905869 28931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:18.907629 28923 server_base.cc:1034] running on GCE node
I20260928 10:57:18.908620 28923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:18.910359 28923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:18.911679 28923 hybrid_clock.cc:648] HybridClock initialized: now 1790593038911585 us; error 70 us; skew 500 ppm
I20260928 10:57:18.912164 28923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:18 dist-test-slave-8w8g krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:18.916409 28923 init.cc:377] Logged in from keytab as kudu/127.5.227.189@KRBTEST.COM (short username kudu)
I20260928 10:57:18.918185 28923 webserver.cc:460] Webserver started at http://127.5.227.189:34419/ using document root <none> and password file <none>
I20260928 10:57:18.918974 28923 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:18.919106 28923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:18.919481 28923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:18.923132 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data/instance:
uuid: "baf71fefde594f4392c1a281f5d61c59"
format_stamp: "Formatted at 2026-09-28 10:57:18 on dist-test-slave-8w8g"
server_key: "894e8cb41adcbdea657e100b63428397"
server_key_iv: "1b84dc9e79eaf3e92c0193ba105aa28e"
server_key_version: "encryptionkey@0"
I20260928 10:57:18.924263 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/wal/instance:
uuid: "baf71fefde594f4392c1a281f5d61c59"
format_stamp: "Formatted at 2026-09-28 10:57:18 on dist-test-slave-8w8g"
server_key: "894e8cb41adcbdea657e100b63428397"
server_key_iv: "1b84dc9e79eaf3e92c0193ba105aa28e"
server_key_version: "encryptionkey@0"
I20260928 10:57:18.931583 28923 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:57:18.935951 28938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:18.937733 28923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:57:18.937916 28923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/wal
uuid: "baf71fefde594f4392c1a281f5d61c59"
format_stamp: "Formatted at 2026-09-28 10:57:18 on dist-test-slave-8w8g"
server_key: "894e8cb41adcbdea657e100b63428397"
server_key_iv: "1b84dc9e79eaf3e92c0193ba105aa28e"
server_key_version: "encryptionkey@0"
I20260928 10:57:18.938238 28923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:18.966018 28923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:18.971328 28923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:18.971782 28923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:18.992890 28923 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.189:35523
I20260928 10:57:18.992915 28989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.189:35523 every 8 connection(s)
I20260928 10:57:18.994443 28923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/data/info.pb
I20260928 10:57:18.996654 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28923
I20260928 10:57:18.996891 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-1/wal/instance
I20260928 10:57:18.997704 6030 external_mini_cluster.cc:1428] Setting key a364a69e30f697c04f543a214968a9bd
I20260928 10:57:19.001165 28990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:19.011590 28990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.227.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.227.188@KRBTEST.COM" created.
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:57:19.041673 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.015063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.5.227.190:39835)
0928 10:57:19.015638 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.015642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.017048 (+ 1406us) server_negotiation.cc:407] Connection header received
0928 10:57:19.019445 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.019452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.019570 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.019721 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.021734 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.022865 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.024756 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.025201 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.025351 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.031960 (+ 6609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.031966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.031969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.032016 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.035632 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.037649 (+ 2017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.037655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.037657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.037833 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.038705 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.038710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.038713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.039080 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.039238 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.040727 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.041104 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":312,"threads_started":1}
I20260928 10:57:19.042946 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.015719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39835 (local address 127.0.0.1:36074)
0928 10:57:19.016808 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.016854 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.019056 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.020071 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.020082 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.020741 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.021438 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.021463 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.023124 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.023128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.024560 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.024577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.025481 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.025489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.025766 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.026571 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.026597 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.031323 (+ 4726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.035980 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.035997 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.036024 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.037324 (+ 1300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.037989 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.037996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.038010 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.038479 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.039283 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.039295 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.039913 (+ 618us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.040593 (+ 680us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.041277 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":321,"threads_started":1}
I20260928 10:57:19.045594 28990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:57:19.058537 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } attempt: 5
I20260928 10:57:19.069074 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.046583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45388)
0928 10:57:19.046818 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.046836 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.046952 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.048235 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.048242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.048820 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.049054 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.049071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.050689 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.050697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.052476 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.052500 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.054014 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.054023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.054274 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.055314 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.055344 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.058847 (+ 3503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.064373 (+ 5526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.064380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.064384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.065237 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.066287 (+ 1050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.066291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.066294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.066530 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.067858 (+ 1328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.067864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.068128 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.068302 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.068633 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 10:57:19.069188 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.046883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.5.227.189:35523)
0928 10:57:19.047612 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.047620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.047650 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:57:19.047707 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.047713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.048045 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.048194 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.049315 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.050501 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.052721 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.053797 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.053956 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.059237 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.059263 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.059286 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.059360 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.064222 (+ 4862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.065630 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.065645 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.065647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.065934 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.066880 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.066887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.066890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.067695 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.068039 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.068444 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.068795 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":568,"thread_start_us":435,"threads_started":1}
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:57:19.073436 28990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:19.075449 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.074109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44376)
0928 10:57:19.074738 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.074935 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab.
Entry for principal kudu/127.5.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab.
W20260928 10:57:19.075973 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.077540 6030 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
/tmp/dist-test-tasko9h9KU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:44877
--webserver_interface=127.5.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab
--principal=kudu/127.5.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.227.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.227.190:39835,127.5.227.189:35523,127.5.227.188:44877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:57:19.077961 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.077522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44386)
0928 10:57:19.077698 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.077796 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 10:57:19.078365 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.082811 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.059302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45394)
0928 10:57:19.059654 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.059678 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.059844 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.061162 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.061166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.061747 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.062005 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.062018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.063897 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.063907 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.065705 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.065723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.066733 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.066741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.067014 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.067981 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.068002 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.071876 (+ 3874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.078758 (+ 6882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.078764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.078767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.079204 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.080182 (+ 978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.080186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.080189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.080366 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.081553 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.081562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.081780 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.081948 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.082371 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260928 10:57:19.083021 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.059523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.5.227.189:35523)
0928 10:57:19.060491 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.060495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.060516 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:57:19.060582 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.060585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.060701 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.060835 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.062217 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.063659 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.066015 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.066552 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.066696 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.072286 (+ 5590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.072292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.072296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.072335 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.078223 (+ 5888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.079579 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.079595 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.079603 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.079889 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.080524 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.080533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.080539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.081159 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.081502 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.082108 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.082386 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":610,"threads_started":1}
I20260928 10:57:19.084405 28916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:19.085546 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.085104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44400)
0928 10:57:19.085293 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.085384 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 10:57:19.086710 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.086200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44406)
0928 10:57:19.086428 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.086540 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 10:57:19.086985 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.127837 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 1
I20260928 10:57:19.129400 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.128703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44414)
0928 10:57:19.128989 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.129150 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 10:57:19.131081 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.130306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44426)
0928 10:57:19.130773 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.130889 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290}
W20260928 10:57:19.131414 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.138280 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 1
I20260928 10:57:19.139684 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.139174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44436)
0928 10:57:19.139402 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.139505 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 10:57:19.140832 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.140481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44444)
0928 10:57:19.140641 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.140723 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 10:57:19.141110 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.199757 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 2
I20260928 10:57:19.201138 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.200579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44458)
0928 10:57:19.200868 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.200970 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260928 10:57:19.202487 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.202000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44462)
0928 10:57:19.202217 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.202323 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260928 10:57:19.202754 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.209389 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 2
I20260928 10:57:19.210677 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.210162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44474)
0928 10:57:19.210421 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.210517 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 10:57:19.212704 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.212169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44480)
0928 10:57:19.212361 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.212557 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 10:57:19.212984 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.294194 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 3
I20260928 10:57:19.295792 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.295017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44490)
0928 10:57:19.295434 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.295551 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 10:57:19.297442 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.296756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44496)
0928 10:57:19.297116 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.297244 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260928 10:57:19.297933 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.319228 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 3
I20260928 10:57:19.320880 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.320191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44502)
0928 10:57:19.320501 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.320633 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 10:57:19.322285 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.321775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44510)
0928 10:57:19.321990 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.322112 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 10:57:19.322575 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.441226 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 4
I20260928 10:57:19.443007 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.442096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44512)
0928 10:57:19.442578 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.442773 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260928 10:57:19.444967 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.444334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44518)
0928 10:57:19.444688 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.444808 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
W20260928 10:57:19.445441 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
W20260928 10:57:19.480733 28999 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:57:19.481245 28999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:57:19.481364 28999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:57:19.481523 28999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:57:19.490937 28999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:57:19.491073 28999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:57:19.491140 28999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:57:19.491256 28999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:57:19.491375 28999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:57:19.499918 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 4
I20260928 10:57:19.501446 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.500807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44532)
0928 10:57:19.501105 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.501214 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 10:57:19.502790 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.502292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44536)
0928 10:57:19.502509 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.502603 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 10:57:19.503077 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.227.188:44877: Network error: Client connection negotiation failed: client connection to 127.5.227.188:44877: connect: Connection refused (error 111)
I20260928 10:57:19.503607 28999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.227.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.227.190:39835,127.5.227.189:35523,127.5.227.188:44877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/krb5kdc/kudu_127.5.227.188.keytab
--principal=kudu/127.5.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.227.188:44877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:57:19.505381 28999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:57:19.507081 28999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:57:19.528733 29004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:19.530074 29005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:57:19.530090 29007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:57:19.531095 28999 server_base.cc:1034] running on GCE node
I20260928 10:57:19.532253 28999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:57:19.534093 28999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:57:19.537771 28999 hybrid_clock.cc:648] HybridClock initialized: now 1790593039537600 us; error 82 us; skew 500 ppm
I20260928 10:57:19.538439 28999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:57:19.543119 28999 init.cc:377] Logged in from keytab as kudu/127.5.227.188@KRBTEST.COM (short username kudu)
I20260928 10:57:19.546307 28999 webserver.cc:460] Webserver started at http://127.5.227.188:45417/ using document root <none> and password file <none>
I20260928 10:57:19.547406 28999 fs_manager.cc:359] Metadata directory not provided
I20260928 10:57:19.547580 28999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:57:19.548031 28999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:57:19.551942 28999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data/instance:
uuid: "5a3e9d397cf14bb19ffcaefbf7975be1"
format_stamp: "Formatted at 2026-09-28 10:57:19 on dist-test-slave-8w8g"
server_key: "00feba03c8ed68a108a749b88dbc04d7"
server_key_iv: "094b8f1f55b65681f827f561adda08be"
server_key_version: "encryptionkey@0"
I20260928 10:57:19.553211 28999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/wal/instance:
uuid: "5a3e9d397cf14bb19ffcaefbf7975be1"
format_stamp: "Formatted at 2026-09-28 10:57:19 on dist-test-slave-8w8g"
server_key: "00feba03c8ed68a108a749b88dbc04d7"
server_key_iv: "094b8f1f55b65681f827f561adda08be"
server_key_version: "encryptionkey@0"
I20260928 10:57:19.560671 28999 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260928 10:57:19.566334 29014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:57:19.568753 28999 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:57:19.569026 28999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/wal
uuid: "5a3e9d397cf14bb19ffcaefbf7975be1"
format_stamp: "Formatted at 2026-09-28 10:57:19 on dist-test-slave-8w8g"
server_key: "00feba03c8ed68a108a749b88dbc04d7"
server_key_iv: "094b8f1f55b65681f827f561adda08be"
server_key_version: "encryptionkey@0"
I20260928 10:57:19.569728 28999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:57:19.621321 28999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:57:19.626322 28999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:57:19.626634 28999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:57:19.649632 28999 rpc_server.cc:307] RPC server started. Bound to: 127.5.227.188:44877
I20260928 10:57:19.649649 29065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.227.188:44877 every 8 connection(s)
I20260928 10:57:19.651125 28999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/data/info.pb
I20260928 10:57:19.655222 29066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:57:19.655941 6030 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu as pid 28999
I20260928 10:57:19.656157 6030 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9h9KU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592619723274-6030-0/minicluster-data/master-2/wal/instance
I20260928 10:57:19.656775 6030 external_mini_cluster.cc:1428] Setting key 2ad49029e2c7428b228d6392a7962efd
I20260928 10:57:19.666106 29066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:57:19.688360 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.658746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.5.227.188:44877)
0928 10:57:19.659605 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.659618 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.659663 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:57:19.662083 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.662114 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.662615 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.663049 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.665197 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.666616 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.669008 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.670993 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.671300 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.677375 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.677385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.677398 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.677453 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.682162 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.683230 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.683243 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.683253 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.683664 (+ 411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.684203 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.684210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.684214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.685130 (+ 916us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.685649 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.686222 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.686797 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":399,"threads_started":1}
I20260928 10:57:19.690296 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.669914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.5.227.190:39835)
0928 10:57:19.670905 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.670936 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.672037 (+ 1101us) server_negotiation.cc:407] Connection header received
0928 10:57:19.672115 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.672119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.672196 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.672318 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.673527 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.674249 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.675897 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.676831 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.677002 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.682055 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.682065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.682071 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.682125 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.686393 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.687426 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.687431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.687434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.687617 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.688509 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.688548 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.688551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.688946 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.689099 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.689765 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.690000 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":619,"threads_started":1}
I20260928 10:57:19.690382 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.670011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.190:39835 (local address 127.0.0.1:36078)
0928 10:57:19.671680 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.671758 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.671917 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.672397 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.672405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.673070 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.673336 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.673346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.674416 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.674430 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.675557 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.675573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.677541 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.677554 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.677752 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.678514 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.678546 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.681742 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.686569 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.686579 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.686586 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.687197 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.687952 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.687961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.687964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.688124 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.689096 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.689105 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.689335 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.689537 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.689893 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"thread_start_us":1038,"threads_started":1}
I20260928 10:57:19.692592 29066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.190@KRBTEST.COM
I20260928 10:57:19.712040 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.693587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.5.227.189:35523)
0928 10:57:19.694151 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.694155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.694171 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:57:19.694219 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.694222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.694286 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.694373 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.695154 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.696039 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.697673 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.698423 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.698540 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.703376 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.703387 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.703394 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.703464 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.707546 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.708892 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.708898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.708902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.709155 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.709715 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.709723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.709729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.710174 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.710311 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.711209 (+ 898us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.711685 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":363,"threads_started":1}
I20260928 10:57:19.712049 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.693628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.189:35523 (local address 127.0.0.1:45396)
0928 10:57:19.693842 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.693863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.693974 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.694445 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.694452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.694857 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.694998 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.695006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.696176 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.696183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.697455 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.697469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.698946 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.698953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.699160 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.699850 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.699876 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.703068 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.707823 (+ 4755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.707842 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.707846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.708626 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.709363 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.709367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.709369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.709554 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.710410 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.710418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.710733 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.710937 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.711336 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 10:57:19.713665 29066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:57:19.713909 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.692218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.5.227.190:39835)
0928 10:57:19.692462 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.692466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.692484 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:57:19.692685 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.692690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.692769 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.692971 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.694523 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.695347 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.697064 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.697833 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.698037 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.704324 (+ 6287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.704339 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.704347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.704553 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.708690 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.710147 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.710155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.710158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.710527 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.711707 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.711716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.711721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.712238 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.712559 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.713242 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.713513 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.188@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.227.189@KRBTEST.COM
I20260928 10:57:19.730818 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.714533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44556)
0928 10:57:19.714816 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.714840 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.715011 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.715907 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.715915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.716636 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.716819 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.716827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.717864 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.717869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.718746 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.718757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.719474 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.719480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.719713 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.720568 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.720586 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.723652 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.728011 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.728017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.728021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.728433 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.728873 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.728878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.728882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.729011 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.729574 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.729579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.729722 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.729890 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.730220 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":27}
I20260928 10:57:19.730870 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.714739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.5.227.188:44877)
0928 10:57:19.714906 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.714910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.714926 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:57:19.715179 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.715184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.715257 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.715351 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.717007 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.717716 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.718901 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.719309 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.719459 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.723953 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.723959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.723963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.723997 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.727805 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.728580 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.728589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.728593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.728739 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.729122 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.729130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.729133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.729454 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.729614 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.729978 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.730386 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":56}
I20260928 10:57:19.734826 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.716255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.5.227.189:35523)
0928 10:57:19.716455 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.716461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.717072 (+ 611us) server_negotiation.cc:407] Connection header received
0928 10:57:19.717272 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.717277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.717359 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.717471 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.718892 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.719702 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.721222 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.721689 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.721843 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.727645 (+ 5802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.727651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.727655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.727699 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.730834 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.732283 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.732289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.732293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.732655 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.733193 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.733199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.733203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.733536 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.733868 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.734204 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.734503 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:57:19.737656 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: Bootstrap starting.
I20260928 10:57:19.741640 6030 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:57:19.742080 29066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:19.743991 29066 log.cc:824] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:19.744740 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 5
I20260928 10:57:19.748553 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: No bootstrap required, opened a new log
I20260928 10:57:19.746031 28891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "91c9720cc44c429c96370690be6aa1e9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36084
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.189@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
I20260928 10:57:19.761996 29066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.762645 29066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3e9d397cf14bb19ffcaefbf7975be1, State: Initialized, Role: FOLLOWER
I20260928 10:57:19.763286 29066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.764571 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.745496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44564)
0928 10:57:19.745744 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.745769 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.745902 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.748325 (+ 2423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.748331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.749005 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.749192 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.749204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.750358 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.750366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.751341 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.751455 (+ 114us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.752481 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.752488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.752720 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.753389 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.753409 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.756171 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.760778 (+ 4607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.760785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.760789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.761435 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.761910 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.761914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.761920 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.762096 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.763137 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.763145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.763339 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.763684 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.764125 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 10:57:19.764887 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.745664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.5.227.188:44877)
0928 10:57:19.747998 (+ 2334us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.748004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.748024 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:57:19.748078 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.748082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.748143 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.748278 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.749371 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.750170 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.751643 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.752289 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.752442 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.756417 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.756424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.756429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.756488 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.760543 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.761597 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.761603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.761606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.761764 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.762225 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.762229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.762232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.762956 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.763145 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.763918 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.764220 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2154}
I20260928 10:57:19.765198 29074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:19.765697 29074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:19.766157 29066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:19.772379 28990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: Bootstrap starting.
I20260928 10:57:19.778599 28990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:19.779716 28990 log.cc:824] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:19.780319 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } attempt: 5
I20260928 10:57:19.787035 28990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: No bootstrap required, opened a new log
Sep 28 10:57:19 dist-test-slave-8w8g krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790593038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.227.190@KRBTEST.COM for kudu/127.5.227.188@KRBTEST.COM
W20260928 10:57:19.795147 29085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:57:19.795300 29085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:57:19.800933 28990 raft_consensus.cc:348] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.801420 28990 raft_consensus.cc:729] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf71fefde594f4392c1a281f5d61c59, State: Initialized, Role: FOLLOWER
I20260928 10:57:19.801977 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.781098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.227.188:44877 (local address 127.0.0.1:44574)
0928 10:57:19.781632 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 10:57:19.781661 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 10:57:19.781792 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:57:19.782656 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:57:19.782662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:57:19.783280 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:57:19.783543 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.783559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.784831 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.784839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.786556 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:57:19.786582 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.788248 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.788256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:57:19.789002 (+ 746us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.789858 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:57:19.789889 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:57:19.793813 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:57:19.798756 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.798764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.798769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.799372 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.799898 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:57:19.799903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:57:19.799908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:57:19.800114 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:57:19.800885 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:57:19.800892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:57:19.801090 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 10:57:19.801244 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 10:57:19.801579 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260928 10:57:19.802028 28990 consensus_queue.cc:260] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.803368 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.781597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.5.227.188:44877)
0928 10:57:19.782057 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.782063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.782083 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:57:19.782145 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.782148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.782215 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.782642 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.783710 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.784647 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.786811 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.787833 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.787976 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.794186 (+ 6210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.794193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.794199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.794244 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.798489 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.799550 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.799557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.799562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.799750 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.800253 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.800258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.800261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.800663 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.800805 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.802344 (+ 1539us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.802768 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260928 10:57:19.804286 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:19.804768 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:19.806381 28990 sys_catalog.cc:564] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:19.810264 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9: Bootstrap starting.
I20260928 10:57:19.815254 28916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9: Neither blocks nor log segments found. Creating new log.
I20260928 10:57:19.816926 28916 log.cc:824] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9: Log is configured to *not* fsync() on all Append() calls
I20260928 10:57:19.780925 28999 master_runner.cc:186] Error getting master registration for 127.5.227.190:39835: OK, instance_id { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" instance_seqno: 1790593038326087 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:57:19.821321 28999 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:57:19.823349 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9: No bootstrap required, opened a new log
W20260928 10:57:19.838140 29100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:57:19.838449 29100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:57:19.834657 28923 master_runner.cc:186] Error getting master registration for 127.5.227.190:39835: OK, instance_id { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" instance_seqno: 1790593038326087 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:57:19.847466 28923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:57:19.851081 28916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.851701 28916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c9720cc44c429c96370690be6aa1e9, State: Initialized, Role: FOLLOWER
I20260928 10:57:19.852378 28916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.854102 29102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:19.854977 29102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:19.855160 28916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:57:19.855223 28891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 10:57:19.855609 28891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:19.859297 28891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.862025 28965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c9720cc44c429c96370690be6aa1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "baf71fefde594f4392c1a281f5d61c59"
I20260928 10:57:19.862675 28965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:19.867111 28965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c9720cc44c429c96370690be6aa1e9 in term 1.
I20260928 10:57:19.868395 28867 leader_election.cc:302] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c9720cc44c429c96370690be6aa1e9, baf71fefde594f4392c1a281f5d61c59; no voters:
I20260928 10:57:19.870229 29041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c9720cc44c429c96370690be6aa1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1"
I20260928 10:57:19.870776 29041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:57:19.874042 28891 leader_election.cc:288] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [CANDIDATE]: Term 1 election: Requested vote from peers baf71fefde594f4392c1a281f5d61c59 (127.5.227.189:35523), 5a3e9d397cf14bb19ffcaefbf7975be1 (127.5.227.188:44877)
I20260928 10:57:19.875797 29041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c9720cc44c429c96370690be6aa1e9 in term 1.
W20260928 10:57:19.879120 29020 tablet.cc:2369] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:19.882382 29102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:57:19.892408 29102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [term 1 LEADER]: Becoming Leader. State: Replica: 91c9720cc44c429c96370690be6aa1e9, State: Running, Role: LEADER
I20260928 10:57:19.893270 29102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } }
I20260928 10:57:19.905586 29108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c9720cc44c429c96370690be6aa1e9. Latest consensus state: current_term: 1 leader_uuid: "91c9720cc44c429c96370690be6aa1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:19.906263 29108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:19.911070 29118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:57:19.935688 29118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:57:19.947084 29124 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:57:19.957080 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.916763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.5.227.189:35523)
0928 10:57:19.916979 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.916988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.924761 (+ 7773us) server_negotiation.cc:407] Connection header received
0928 10:57:19.925168 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.925174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.925271 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.925527 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.937506 (+ 11979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.938601 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.940513 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.941149 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.941277 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.945398 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.945407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.945411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.945454 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.952841 (+ 7387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.954029 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.954036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.954041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.954230 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.954698 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.954703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.954706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.955122 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.956303 (+ 1181us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.956430 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.956686 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:57:19.957086 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.925033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.5.227.188:44877)
0928 10:57:19.925270 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.925277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.933992 (+ 8715us) server_negotiation.cc:407] Connection header received
0928 10:57:19.934313 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.934320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.934520 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.934612 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.936967 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.938126 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.940154 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.940884 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.940989 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.945209 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.945219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.945225 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.945279 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.950613 (+ 5334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.954297 (+ 3684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.954304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.954307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.954528 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.954987 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.954992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.954996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.955414 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.955689 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.956395 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.956686 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:57:19.973066 29041 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [term 1 FOLLOWER]: Refusing update from remote peer 91c9720cc44c429c96370690be6aa1e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:57:19.974604 28965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [term 1 FOLLOWER]: Refusing update from remote peer 91c9720cc44c429c96370690be6aa1e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 10:57:19.975420 28944 tablet.cc:2369] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:57:19.976513 29102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:19.977388 29102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:57:19.979012 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:19.915638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.5.227.190:39835)
0928 10:57:19.915962 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:19.915973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:19.924720 (+ 8747us) server_negotiation.cc:407] Connection header received
0928 10:57:19.925199 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:19.925207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:19.925304 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:19.925398 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:19.944602 (+ 19204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.945766 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.952053 (+ 6287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:19.952786 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:19.952895 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:19.964001 (+ 11106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:57:19.964010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:57:19.964018 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:57:19.964118 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:57:19.968399 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.969497 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.969503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.969508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.969673 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:57:19.970134 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:57:19.970139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:57:19.970143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:57:19.972812 (+ 2669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:57:19.973121 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:19.974419 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0928 10:57:19.974732 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:57:20.004722 29074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c9720cc44c429c96370690be6aa1e9. Latest consensus state: current_term: 1 leader_uuid: "91c9720cc44c429c96370690be6aa1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:20.005133 29074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:20.015228 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c9720cc44c429c96370690be6aa1e9. Latest consensus state: current_term: 1 leader_uuid: "91c9720cc44c429c96370690be6aa1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:20.015592 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:20.016259 29102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c9720cc44c429c96370690be6aa1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:20.016618 29102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:20.018556 29118 catalog_manager.cc:1347] Generated new cluster ID: fccfef479f704363af83c8aa1acd5440
I20260928 10:57:20.018676 29118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:57:20.020262 29074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c9720cc44c429c96370690be6aa1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:20.020663 29074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:20.032168 29102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c9720cc44c429c96370690be6aa1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:20.032500 29102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9 [sys.catalog]: This master's current role is: LEADER
I20260928 10:57:20.041415 29126 mvcc.cc:205] Tried to move back new op lower bound from 7334269091711209472 to 7334269091423617024. Current Snapshot: MvccSnapshot[applied={T|T < 7334269091711209472}]
I20260928 10:57:20.056437 29118 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:57:20.059038 29118 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:57:20.062098 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c9720cc44c429c96370690be6aa1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c9720cc44c429c96370690be6aa1e9" member_type: VOTER last_known_addr { host: "127.5.227.190" port: 39835 } } peers { permanent_uuid: "baf71fefde594f4392c1a281f5d61c59" member_type: VOTER last_known_addr { host: "127.5.227.189" port: 35523 } } peers { permanent_uuid: "5a3e9d397cf14bb19ffcaefbf7975be1" member_type: VOTER last_known_addr { host: "127.5.227.188" port: 44877 } } }
I20260928 10:57:20.062388 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:57:20.092909 29118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91c9720cc44c429c96370690be6aa1e9: Generated new TSK 0
I20260928 10:57:20.094229 29118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:57:20.258244 29070 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.230539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.5.227.190:39835)
0928 10:57:20.231363 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.231370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.253069 (+ 21699us) server_negotiation.cc:407] Connection header received
0928 10:57:20.255277 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.255284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.255384 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.255483 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.256782 (+ 1299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36098: BlockingRecv error: recv got EOF from 127.0.0.1:36098 (error 108)
Metrics: {"server-negotiator.queue_time_us":599}
W20260928 10:57:20.261703 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.237615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.5.227.189:35523)
0928 10:57:20.237818 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.237824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.259496 (+ 21672us) server_negotiation.cc:407] Connection header received
0928 10:57:20.260043 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.260049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.260138 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.260218 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.261417 (+ 1199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45408: BlockingRecv error: recv got EOF from 127.0.0.1:45408 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 10:57:20.265059 29069 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.253765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.5.227.188:44877)
0928 10:57:20.253973 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.253979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.262627 (+ 8648us) server_negotiation.cc:407] Connection header received
0928 10:57:20.263103 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.263111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.263197 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.263281 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.264787 (+ 1506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44584: BlockingRecv error: recv got EOF from 127.0.0.1:44584 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 10:57:20.311089 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.296001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.5.227.189:35523)
0928 10:57:20.296222 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.296229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.308504 (+ 12275us) server_negotiation.cc:407] Connection header received
0928 10:57:20.309095 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.309109 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.309247 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.309350 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.310771 (+ 1421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45414: BlockingRecv error: recv got EOF from 127.0.0.1:45414 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 10:57:20.366355 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.356619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.5.227.189:35523)
0928 10:57:20.356809 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.356815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.364434 (+ 7619us) server_negotiation.cc:407] Connection header received
0928 10:57:20.364761 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.364768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.364885 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.364963 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.366084 (+ 1121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59540: BlockingRecv error: recv got EOF from 127.0.0.1:59540 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 10:57:20.389698 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.374455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.5.227.189:35523)
0928 10:57:20.374689 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.374699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.387253 (+ 12554us) server_negotiation.cc:407] Connection header received
0928 10:57:20.387944 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.387951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.388063 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.388154 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.389377 (+ 1223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59550: BlockingRecv error: recv got EOF from 127.0.0.1:59550 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 10:57:20.444051 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.433132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.5.227.189:35523)
0928 10:57:20.433448 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.433455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.441088 (+ 7633us) server_negotiation.cc:407] Connection header received
0928 10:57:20.441679 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.441686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.441800 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.441888 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.443391 (+ 1503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59564: BlockingRecv error: recv got EOF from 127.0.0.1:59564 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260928 10:57:20.499478 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.489230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.5.227.189:35523)
0928 10:57:20.489436 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.489443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.497124 (+ 7681us) server_negotiation.cc:407] Connection header received
0928 10:57:20.497744 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.497750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.497888 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.497976 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.499167 (+ 1191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59572: BlockingRecv error: recv got EOF from 127.0.0.1:59572 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 10:57:20.558917 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.551935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.5.227.189:35523)
0928 10:57:20.552130 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.552136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.557247 (+ 5111us) server_negotiation.cc:407] Connection header received
0928 10:57:20.557462 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.557468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.557567 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.557688 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.558639 (+ 951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59582: BlockingRecv error: recv got EOF from 127.0.0.1:59582 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 10:57:20.597258 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.595153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.5.227.189:35523)
0928 10:57:20.595443 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.595451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.595752 (+ 301us) server_negotiation.cc:407] Connection header received
0928 10:57:20.596061 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.596065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.596142 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.596258 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.597012 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59584: BlockingRecv error: recv got EOF from 127.0.0.1:59584 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 10:57:20.667971 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.665579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.5.227.189:35523)
0928 10:57:20.665794 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.665802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.666245 (+ 443us) server_negotiation.cc:407] Connection header received
0928 10:57:20.666502 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.666507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.666598 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.666723 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.667676 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59596: BlockingRecv error: recv got EOF from 127.0.0.1:59596 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:57:20.797819 29085 catalog_manager.cc:1259] Loaded cluster ID: fccfef479f704363af83c8aa1acd5440
I20260928 10:57:20.797968 29085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: loading cluster ID for follower catalog manager: success
I20260928 10:57:20.801199 29085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: acquiring CA information for follower catalog manager: success
W20260928 10:57:20.802501 29072 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:57:20.800265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.5.227.189:35523)
0928 10:57:20.800489 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:20.800496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:20.800867 (+ 371us) server_negotiation.cc:407] Connection header received
0928 10:57:20.801125 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:20.801130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:20.801204 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:20.801298 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:57:20.802252 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59606: BlockingRecv error: recv got EOF from 127.0.0.1:59606 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 10:57:20.804103 29085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a3e9d397cf14bb19ffcaefbf7975be1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:57:20.840725 29100 catalog_manager.cc:1259] Loaded cluster ID: fccfef479f704363af83c8aa1acd5440
I20260928 10:57:20.840865 29100 catalog_manager.cc:1552] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: loading cluster ID for follower catalog manager: success
I20260928 10:57:20.843724 29100 catalog_manager.cc:1574] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: acquiring CA information for follower catalog manager: success
I20260928 10:57:20.845764 29100 catalog_manager.cc:1602] T 00000000000000000000000000000000 P baf71fefde594f4392c1a281f5d61c59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:57:21.071257 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:21.063832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.5.227.189:35523)
0928 10:57:21.064042 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:21.064049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:21.064453 (+ 404us) server_negotiation.cc:407] Connection header received
0928 10:57:21.064702 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:21.064708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:21.064885 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:21.065045 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:21.065576 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.066441 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.068140 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.068674 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.068806 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:21.069636 (+ 830us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:21.070393 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:21.070512 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:21.070724 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 10:57:21.070920 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:57:21.085897 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:21.078183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.5.227.188:44877)
0928 10:57:21.078788 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:21.078793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:21.078813 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:57:21.079133 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:21.079137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:21.079335 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:21.079512 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:21.080122 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.081009 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.082746 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.083360 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.083516 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:21.084392 (+ 876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:21.085014 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:21.085153 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:21.085399 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 10:57:21.085496 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":345,"threads_started":1}
I20260928 10:57:21.104568 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:21.094748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.5.227.190:39835)
0928 10:57:21.096223 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:21.096229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:21.096250 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:57:21.096382 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:21.096389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:21.096634 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:21.097108 (+ 474us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:21.098219 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.099279 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.101520 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.102123 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.102277 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:21.103051 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:21.103490 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:21.103612 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:21.104129 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 10:57:21.104230 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":1158,"threads_started":1}
I20260928 10:57:21.104568 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:21.094748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.5.227.189:35523)
0928 10:57:21.095014 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:21.095024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:21.096490 (+ 1466us) server_negotiation.cc:407] Connection header received
0928 10:57:21.096806 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:21.096812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:21.096928 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:21.097007 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:21.098324 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.099737 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.101806 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.102488 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.102633 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:21.103333 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:21.103685 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:21.103833 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:21.104112 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 10:57:21.104218 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:57:21.104914 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:57:21.096768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.5.227.188:44877)
0928 10:57:21.096992 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:57:21.097001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:57:21.097904 (+ 903us) server_negotiation.cc:407] Connection header received
0928 10:57:21.098099 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:57:21.098107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:57:21.098270 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:57:21.098578 (+ 308us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:57:21.099087 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.100201 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.101894 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:57:21.102529 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:57:21.102642 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:57:21.103829 (+ 1187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:57:21.103962 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:57:21.104054 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 10:57:21.104516 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 10:57:21.104615 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:57:21.113276 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28849
I20260928 10:57:21.114923 28911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:21.244542 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28849
W20260928 10:57:21.267354 29016 connection.cc:535] server connection from 127.0.0.1:44574 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:57:21.267721 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28923
I20260928 10:57:21.269044 28985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:21.386355 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28923
I20260928 10:57:21.409281 6030 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu pid 28999
I20260928 10:57:21.411231 29061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:57:21.524158 6030 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9h9KU/build/asan/bin/kudu with pid 28999
2026-09-28T10:57:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5387 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11146 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (421754 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST